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A  Summary  of  Flightdeck  Observer  Data 
From  SafeFlight  21  OpEval-2 


INTRODUCTION 

The  availability  of  new  technologies  for  the 
flightdeck  and  air  traffic  control  facilities  is  creating 
new  capabilities  for  enhanced  aircraft  operations  and, 
with  them,  the  need  to  evaluate  the  effectiveness  of 
these  new  technologies  in  operational  settings.  Two 
such  systems.  Cockpit  Display  of  Traffic  Information 
(CDTI)  and  Automatic  Dependent  Surveillance- 
Broadcast  (ADS-B),  were  recently  demonstrated  in 
Operational  Evaluation  2  (OpEval-2),  which  was 
conducted  at  Standiford  International  Airport  in 
Louisville,  KY,  OpEval-2  was  sponsored  by  the  Cargo 
Airlines  Association  (CAA)  and  the  SafeFlight21  Of¬ 
fice  of  the  Federal  Aviation  Administration  (FAA), 
and  included  aircraft  and  flight  crews  from  industry, 
government,  and  private  organizations. 

The  purpose  of  OpEval-2  was  to  demonstrate  the 
use  of  CDTI  and  ADS-B  and  their  expected  benefits, 
which  include  increased  safety,  enhanced  capacity, 
and  greater  efficiency.  Three  objectives  were  planned 
for  OpEval-2.  The  first  objective  was  to  develop  and 
evaluate  avionics  and  procedural  modifications  needed 
to  support  operational  approval  for  the  following 
applications:  Initial  and  Final  Approach  Spacing, 
Departure  Spacing,  Final  Approach  and  Runway 
Occupancy  Awareness  (FAROA),  and  Airport  Sur¬ 
face  Situation  Awareness  (ASSA).  A  fifth  application. 
Visual  Acquisition  and  Traffic  Awareness,  was  tested 
in  OpEval-1,  but  also  was  included  in  OpEval-2  to 
evaluate  a  specific  procedure  (see  OpEval-1  Final 
Report,  1999,  for  OpEval-1  summary).  The  visual 
acquisition  application  included  the  use  of  call  sign  in 
the  traffic  call-out  to  take  advantage  of  the  informa¬ 
tion  available  on  the  traffic  display.  The  second  objec¬ 
tive  was  to  use  these  applications  to  evaluate  ADS-B 
technology  in  a  terminal  area  environment  with  air 
traffic  controllers.  The  third  objective  was  to  high¬ 
light  the  safety  and  efficiency  benefits  of  ADS-B  by 
providing  a  limited  demonstration  to  key  industry 
participants,  including  labor,  airline  operations,  gen¬ 
eral  aviation,  and  the  FAA. 

An  important  part  of  OpEval-2  was  the  collection 
and  analysis  of  flightdeck  observer  data  to  aid  in 
quantifying  and  verifying  the  expected  benefits  of 
CDTI.  This  report  summarizes  the  development  of 


flightdeck  observer  data  forms  and  flight  crew  ques¬ 
tionnaires,  the  training  of  flightdeck  observers,  and 
the  collection  and  analysis  of  flightdeck  observer  data 
for  OpEval-2.  In  addition,  a  summary  of  human 
factors  issues  related  to  CDTI  is  presented  for  each 
OpEval-2  application  based  on  the  results  of  flightdeck 
observer  data  analysis.  Before  these  summaries  are 
presented,  a  brief  overview  of  OpEval-2  procedures 
and  applications  is  given. 

OPEVAL-2  PROCEDURES 

Several  procedures  were  implemented  to  ensure  the 
safety  of  flight  operations  at  Standiford  International 
Airport  during  OpEval-2,  which  consisted  of  six, 
three-hour  flight  periods  that  occurred  between  Oc¬ 
tober  26th  and  October  30th  of  2000.  No  observer 
data  was  collected  during  the  sixth  flight  period, 
which  comprised  a  limited  demonstration  to  key 
industry  participants.  This  flight  period  is  not  dis¬ 
cussed  further.  The  flight  periods  were  timed  to 
ensure  that  OpEval-2  occurred  during  low  traffic 
volume  at  the  airport.  A  separate  team  of  eight  con¬ 
trollers,  including  a  coordinator  for  both  the  Air 
Traffic  Control  Tower  (ATCT)  and  Terminal  Radar 
Approach  Control  (TRACON),  was  trained  to  pro¬ 
vide  standard  air  traffic  control  sendees  for  all  OpEval- 
2  aircraft. 

The  ATCT  limited  access  to  runway  17R/35L  to 
only  the  16  aircraft  participating  in  OpEval-2.  In 
addition,  the  ATCT  isolated  the  airspace  near  the 
airport  for  OpEval-2  air  traffic  to  minimize  disrup¬ 
tion  to  normal  flight  arrivals  and  departures  that  used 
runway  17L/35R.  The  controllers  established  two 
traffic  patterns  for  OpEval-2.  The  first  pattern  was 
labeled  Outer  Covey,  and  it  was  restricted  to  the  jet 
aircraft  that  were  participating  in  OpEval-2.  The 
second  pattern  was  labeled  Inner  Covey,  and  it  con¬ 
sisted  of  a  wide  variety  of  aircraft  and  equipage, 
including  general  aviation  aircraft.  Traffic  from  these 
two  patterns  was  merged  by  the  controllers  at  a  point 
20  miles  from  the  17R/35L  runway  threshold. 

Table  1  displays  the  participating  aircraft  by  flight 
period,  application,  and  equipment  type.  Surface 
map  capability  is  also  indicated.  The  flight  periods 
were  developed  such  that  the  systematic  variation  of 
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Table  1.  Opeval-2  aircraft  classified  by  Flight  Event,  Application,  and  CDTI  Type. 


|V  •: 

Aircraft  ID 

Flight 
Period  1 

Flight 
Period  2 

Flight 
Period  3 

Flight 
Period  A 

Flight 
Period  5 

ASSA 

FAROA 
Air  and 
Ground 

Departure 

Spacing 

Initial  and 
Final  Spacing 

Visual  Acquisition/ 
Traffic  Awareness 

UPS  1  (Build  4) 
Boeing  727 
UPS101 

/ 

/ 

/ 

/ 

No 

Map 

/ 

/ 

/ 

/ 

UPS  2  (Build  4) 
Boeing  727 
UPS202 

/ 

/ 

/ 

/ 

No 

Map 

/ 

/ 

/ 

/ 

Build  2/A 

UPS  3  (Build  4) 
Boeing  727 
UPS303 

/ 

/ 

/ 

/ 

No 

Map 

/ 

/ 

/ 

UPS  4  (Build  4) 
Boeing  727 
UPS404 

/ 

/ 

/ 

/ 

No 

Map 

/ 

/ 

/ 

FedEx  1  (Build  2) 
Boeing  727 
FDX1 

/ 

/ 

No 

Map 

/ 

/ 

/ 

/ 

& 

1 1 
«  2 

FAA1 

Boeing  727 
N40 

/ 

/ 

/ 

No 

Map 

/ 

/ 

/ 

/ 

FAA2 

Convair  580 
N49 

/ 

/ 

/ 

No 

Map 

/ 

/ 

/ 

/ 

FAA3 

Convair  580 
N39 

/ 

/ 

■ 

/ 

No 

Map 

/ 

/ 

/ 

/ 

Collins1 

Sabreliner 

N50CR 

/ 

■ 

/ 

/ 

/ 

/ 

■a  cj 
at  a. 

c  & 
a  2 

L3 

Cessna  Citation  V 
N189H 

/ 

/ 

/ 

■ 

/ 

/ 

/ 

<  Cu 

/ 

H 

■ 

/ 

1  The  Collins  Sabreliner  had  the  advanced  spacing  algorithm,  but  not  the  Surface  Moving  Map. 

2The  Honeywell  King  Air  had  the  Surface  Moving  Map,  but  not  the  advanced  spacing  algorithm. 

3The  UPS  AT  King  Air  was  equipped  with  both  CDTI  Build2/4  and  MX20  platforms.  CDTI  Build2/4  platform  was  used  during  Flight  Event 
1 .  Both  platform  types  were  used  during  Flight  Event  2. 
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procedures  within  and  across  these  periods  produced 
an  extensive  set  of  data  that  was  used  to  evaluate 
OpEval-2  applications.  The  following  paragraphs 
briefly  summarize  each  of  the  five  flight  periods  In 
which  flightdeck  observers  collected  data. 

Flight  Period  1 

Operations  during  Flight  Period  1  were  conducted 
on  RWY  17R.  The  first  90  minutes  of  flight  time 
included  three  full-stops,  one  after  each  of  the  long 
approach  spacing  profiles  (i.e,,  closing  to  5  miles  or 
120  seconds  at  the  runway  threshold).  Table  1  dis¬ 
plays  the  participating  aircraft  by  flight  period,  appli¬ 
cation,  and  equipment  type.  Surface  map  capability  is 
also  indicated.  Table  1  shows  that  the  three  FAA 
aircraft  participated,  as  did  the  L3  and  Collins  busi¬ 
ness  jets.  The  remaining  90  minutes  of  flight  time 
consisted  of  low  approaches  and  use  of  the  OpEval-2 
traffic  call  sign  procedure,  as  well  as  runway  occu¬ 
pancy  awareness  scenarios.  These  low  approaches 
included  four  GA  (Beech  Bonanza,  Cessna  210,  Piper 
Lance,  Piper  Aztec),  two  Beech  King  Air  (A90,  C90), 
and  two  FAA  Convair  580  aircraft. 

Flight  Period  2 

Operations  during  Flight  Period  2  were  similar  to 
those  of  Flight  Period  1 ,  and  they  also  were  conducted 
on  RWY  17R.  Flight  crews  conducted  two  full-stops, 
one  after  each  of  the  long  approach  spacing  profiles  in 
the  first  90  minutes  of  flight  time,  followed  by  low 
approaches  for  the  remaining  90  minutes.  Table  1 
shows  that  the  same  aircraft  participated  in  Flight 
Periods  1  and  2.  In  addition,  four  UPS  aircraft  per¬ 
formed  three  full-stops,  one  after  each  of  the  long 
approach  spacing  profiles  in  the  first  90  minutes  of 
Flight  Period  2.  Also,  four  UPS  aircraft  performed 
three  long-approach  spacing  profiles  to  a  full  stop 
landing  in  the  first  90  minutes  of  Flight  Period  2. 

Flight  Period  3 

Operations  during  Flight  Period  3  consisted  of  four 
full-stop  sequences  to  RWY  35L.  Long-  approach  spac¬ 
ing  profiles  wereflown  and  operations  were  limited  to 
the  Outer  Covey  pattern.  Table  1  shows  that  only  the 
UPS  and  Federal  Express  aircraft  participated  in  Flight 
Period  3. 

Flight  Period  4 

Operations  during  Flight  Period  4  consisted  of  four 
full-stop  sequences  to  RWY  35L.  Short-  approach  spac¬ 
ing  profiles  (i.e.,  closing  to  3  miles  or  90  seconds  at  the 
runway  threshold)  were  flown  and  operations  again 
were  limited  to  the  Outer  Covey  pattern.  Table  1 


shows  that  the  UPS  and  Federal  Express  aircraft 
participated  in  Flight  Period  4,  along  with  the  L3  and 
Collins  business  jets. 

Flight  Period  5 

Flight  Period  5  operations  were  similar  to  those  of 
Flight  Period  2,  and  they  were  conducted  on  RWY 
35L.  Whereas  In  Flight  Period  2,  flight  crews  con¬ 
ducted  long-approach  spacing  profiles;  for  Flight 
Period  5>  they  conducted  two  full-stop,  short-ap¬ 
proach  spacing  profiles  In  the  first  90  minutes  of 
flight  time,  followed  by  low  approaches  for  the  re¬ 
maining  90  minutes.  Table  1  shows  that  the  same 
aircraft  participated  in  Flight  Periods  2  and  5. 

OPEVAL-2  APPLICATIONS 

The  five  flight  periods  provided  a  means  for  evalu¬ 
ating  each  of  the  OpEvai-2  applications.  The  follow¬ 
ing  paragraphs  briefly  describe  tasks  and  operations 
that  were  developed  and  then  embedded  within  the 
flight  periods  to  allow  for  evaluation  of  each  OpEval- 
2  application. 

Airport  Surface  Situation  Awareness  (ASSA) 

A  set  of  circuitous  taxi  routes  was  designed  to 
evaluate  the  ability  of  flight  crews  to  accurately  navi¬ 
gate  with  either  an  electronic  surface  map  display  or 
a  paper  surface  map.  The  routes  were  created  to  assess 
the  extent  to  which  a  CDTI  system  that  displayed 
airport  surface  information  as  well  as  traffic  could 
Improve  the  situation  awareness  of  flight  crews  during 
airport  surface  operations.  This  evaluation  assessed 
the  flight  crews'  ability  to  identify  specific  geographic 
locations  on  the  airport  surface  and  the  relative  loca¬ 
tion  of  other  aircraft  and  surface  vehicles.  Routes  were 
predefined,  named,  and  provided  to  the  controllers  in 
graphical  format  and  to  the  flight  crews  in  written  text 
format.  Within  each  flight  period,  the  taxi  routes  were 
assigned  to  pre-selected  aircraft  on  either  their  out¬ 
bound  or  inbound  taxi  legs.  The  Ground  controllers 
cleared  aircraft  to  taxi  via  the  route  name,  and  then 
they  monitored  aircraft  movements  to  ensure  compli¬ 
ance  with  the  prescribed  route.  Flight  crews  inter¬ 
preted  the  written  text  as  a  series  of  turn  instructions 
to  determine  the  route  to  taxi.  Flight  crews  were  asked 
to  stop  the  aircraft  at  a  predetermined  point  along  the 
prescribed  route  to  answer  two  questions  posed  by 
observer s.  The  first  question  asked  them  to  identify 
their  current  location  and  the  second  question  asked 
them  to  identify  the  most  proximal  surface  traffic. 
Additional  information  was  recorded  during  normal 
taxi  to  and  from  the  ramp  area. 
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Departure  Spacing 

Aircraft  participating  in  the  Outside  Covey  par¬ 
ticipated  in  the  Departure  Spacing  application,  which 
was  designed  to  evaluate  the  ability  of  the  flight  crew 
and  controllers  to  manage  a  pre-determined  spacing 
interval  between  departing  aircraft.  Longspacing  (i.e., 
6.0  nm)  and  short  spacing  (i.e.,  4.5  nm)  scenarios 
were  defined  for  each  flight  period.  Each  time  the 
Outer  Covey  pattern  was  flown,  flight  crews  deter¬ 
mined  the  departure  spacing  interval  between  all  but 
the  last  aircraft,  for  which  the  local  tower  controller 
provided  the  initial  departure  spacing  interval. 

Two  procedures  were  used  during  OpEval-2  to 
evaluate  the  Departure  Spacing  application  and  en¬ 
sure  compliance  with  standard  ATC  procedures.  The 
first  procedure,  known  as  flight  crew-managed  depar¬ 
ture  spacing,  required  flight  crews  to  use  mission 
reference  cards  that  outlined  the  procedures  to  follow 
and  distances  to  maintain.  As  part  of  this  procedure, 
controllers  referred  to  scenario  cards  that  defined  the 
departure  spacing  interval.  Local  control  provided  a 
take-off  clearance  using  standard  phraseology  when 
standard  departure  separation  was  achieved  from  a 
preceding  departure  (i.e.,  6000  ft.  and  airborne).  The 
flight  crews  then  positioned  the  aircraft  on  the  OpEval- 
2  runway,  waited  until  the  appropriate  distance  was 
achieved,  and  then  began  the  take-off  roll.  ATC 
ensured  the  runway  was  protected  while  the  flight 
crew  was  waiting  for  the  preceding  departure  to  achieve 
the  appropriate  distance. 

In  the  second  procedure,  known  as  controller- 
managed  departure  spacing,  flight  crews  were  re¬ 
quired  to  use  the  mission  reference  cards  and 
controllers  used  the  scenario  cards  that  defined  the  to- 
be-achieved  departure  spacing  interval.  However,  the 
controller  first  placed  the  aircraft  on  the  runway  with 
instructions  to  taxi  into  position  and  hold.  The  con¬ 
troller  issued  a  take-off  clearance  using  standard  phrase¬ 
ology  once  the  aircraft  achieved  the  appropriate 
departure  spacing  distance. 

Initial  and  Final  Approach  Spacing 

The  purpose  of  the  Initial  and  Final  Approach 
Spacing  application  was  to  evaluate  the  ability  of  the 
flight  crew  and  controllers  to  manage  a  pre-deter- 
mined  approach  spacing  interval  between  arriving 
aircraft.  The  required  spacing  changed  as  the  aircraft 
continued  along  the  pattern  and  approached  the  air¬ 
port.  The  final  spacing  between  aircraft  as  the  lead 
aircraft  crossed  the  threshold  was  either  5  nm/ 1 20  sec 
or  3  nm/90  sec  (depending  on  the  flight  period).  The 
scenarios  were  designed  to  evaluate  two  levels  of 


CDTI  approach  spacing  capability  -  an  initial  capa¬ 
bility  (herein  referred  to  as  UPS  AT  Build  2/4)  and  a 
more  advanced  prototype  avionics  capability  that 
included  approach  spacing  algorithms  (herein  re¬ 
ferred  to  as  Advanced  Prototype).  The  traffic  flows 
from  the  Inner  and  Outer  Covey  patterns  were  merged 
and  sequenced  by  ATC  at  20  miles  from  the  runway 
threshold.  The  initial  approach  spacing  at  the  20-mile 
point  was  intended  to  be  6.5  nm  (long  spacing) 
during  some  of  the  flight  periods  and  5.5  nm  (short 
spacing)  during  other  flight  periods.  The  ability  of 
ATC  to  meet  these  distances  depended  on  several 
factors,  including  spacing  at  the  end  of  the  departure 
profile  and  separation  from  non-participating  aircraft 
that  were  inbound  to  the  non-OpEval-2  runway. 
Flight  crews  used  CDTI  during  the  20-mile  approach 
to  reduce  the  spacing  interval  to  a  long  (5  nm  or  120 
sec  if  initial  spacing  was  6.5  nm)  or  a  short  (3  nm  or 
90  sec  if  initial  spacing  was  5.5  nm)  final  approach 
spacing  criteria.  ATC  monitored  both  participating 
and  non-participating  aircraft  and  responded  appro¬ 
priately  to  ensure  that  at  least  minimum  standard 
separation  was  maintained. 

Final  Approach  and  Runway  Occupancy 
Awareness  (FAROA) 

The  FAROA  application  included  use  of  the  CDTI 
during  OpEval-2  departure  and  arrival  operations. 
Flight  crews  were  required  to  use  CDTI  immediately 
prior  to  departures  and  arrivals  to  scan  for  targets  that 
were  positioned  on  or  near  the  runway.  This  proce¬ 
dure  was  used  to  determine  if  CDTI  use  increased 
flight  crew  awareness  of  runway  occupancy.  Occa¬ 
sionally,  an  ADS-B  equipped  surface  vehicle  and/or 
OpEval-2  aircraft  were  required  to  hold  on  the  taxi- 
ways  to  provide  flight  crews  using  CDTI  with  targets 
that  were  off  the  runway.  Three  flight  periods  in¬ 
cluded  scenarios  in  which  the  FAA  B727  and  an  ADS- 
B  equipped  surface  vehicle  were  positioned  on  the 
OpEval-2  runway.  When  a  vehicle  was  on  the  runway, 
flight  crews  were  cleared  to  complete  a  low  approach 
at  500  feet  above  ground  level  (AGL)  and  data  were 
collected  on  whether  the  flight  crews  became  aware  of 
the  conflict  using  CDTI  and  associated  flight  crew 
workload. 

Visual  Acquisition  and  Traffic  Awareness 

The  Visual  Acquisition  and  Traffic  Awareness  ap¬ 
plication  involved  the  use  of  ATC  traffic  advisories 
during  each  of  the  five  flight  periods.  The  availability 
of  aircraft  call  sign  on  CDTI  made  it  possible  to 
collect  data  on  the  use  of  aircraft  call  sign  in  traffic 
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advisories.  Although  this  is  not  standard  AT C  phrase¬ 
ology,  FAA  Order  7110.65  does  allow  additional 
information  to  be  appended  to  traffic  advisories. 
Therefore,  the  flight  profiles  were  designed  to  evalu¬ 
ate  the  inclusion  of  aircraft  call  sign  in  traffic  adviso¬ 
ries  and  replies  by  flight  crews.  Normal  traffic  calls 
were  used  during  the  Departure  and  Approach  Spac¬ 
ing  applications  for  aircraft  in  the  Outer  Covey  pat¬ 
tern.  However,  aircraft  in  the  Inner  Covey  pattern 
were  vectored  by  ATC  to  create  as  many  traffic  calls  as 
possible  during  flight  periods  1,  2  and  5.  This  proce¬ 
dure  was  used  to  assess  the  effect  of  including  aircraft 
call  sign  in  traffic  advisories  when  air  traffic  was 
increased. 

DESCRIPTION  OF  CDTI  FUNCTION 

The  CDTI  displays  the  relative  position  of  proxi¬ 
mal  traffic  to  own  aircraft.  Additional  information 
also  may  be  displayed,  such  as  navigational  aids  and 
obstructions.  The  CDTI  can  obtain  traffic  informa¬ 
tion  from  several  different  sources,  including  ADS-B, 
Traffic  Information  Service  (TIS),  or  an  on-board 
T raffic  Alert  and  Collision  Avoidance  System  (T CAS) . 
Traffic  information  is  displayed  via  a  dedicated  avi¬ 
onics  device  (i.e.,  a  CDTI),  or  the  information  can  be 
displayed  on  a  shared  avionics  device  (i.e.,  multi¬ 
function  display).  The  latter  form  of  display  attempts 
to  integrate  information  from  several  sources  into  a 
single  display. 

Figure  1  shows  the  three  different  CDTI  platforms 
that  were  evaluated  during  OpEval-2.  These  were  the 
UPS  AT  Build  2/4,  the  UPS  AT  MX20,  and  Prototype 
Avionics.  The  UPS  AT  Build  2/4  platform  included 
two  unique  UPS  AT  Link  and  Display  Processor  Unit 
(LDPU)  versions  that  were  identified  as  Build  2  and 
Build  4.  The  Build  2  platform  consisted  of  a  Mode  S 
Transponder,  TIS,  and  a  ADC/ Universal  CDTI  and 


Keyboard,  The  Build  4  platform  consisted  of  both  a 
Mode  S  and  a  UAT  T ransponder,  and  an  Astronautics 
CDTI  and  Keyboard.  The  Build  4  platform  was  a 
newer  (updated)  version  of  the  Build  2  platform.  The 
UPS  AT  MX-20  platform  Included  an  Apollo  GX50 
GPS  receiver,  a  UAT  transceiver,  and  the  Apollo  MX-20 
multi-function  cockpit  display.  Finally,  the  Prototype 
Avionics  platform  consisted  of  an  Advanced  Prototype 
capability  and  a  Basic  Prototype  capability.  The  former 
capability  included  either  a  surface  map  overlay  and/or 
an  algorithm  for  advanced  approach  spacing.  The  Basic 
Prototype  Included  rudimentary  CDTI  capability  and 
was  not  intended  to  be  representative  of  potential  CDTI 
implementation. 

FLIGHTDECK  OBSERVATION  METHOD 

Flight  crew  interaction  with  each  of  the  CDTI 
platforms  was  evaluated  during  the  ASSA,  Departure 
Spacing,  Visual  Acquisition  and  Traffic  Awareness, 
Initial  and  Final  Approach  Spacing,  and  FAROA 
OpEval-2  applications.  As  Table  1  illustrates,  some 
aircraft  were  not  equipped  to  perform  certain  OpEval- 
2  applications;  hence,  flightdeck  observer  data  was 
not  collected  for  these  applications.  Flightdeck  ob¬ 
server  data  aided  in  the  identification  of  human  factors 
and  performance  issues  associated  with  the  use  of  each 
CDTI  display  platform  for  the  OpEval-2  applications. 

Two  methods  of  flightdeck  observer  data  collec¬ 
tion  were  designed  to  capture  flight  crew  interaction 
with  CDTI  platforms.  The  first  method  consisted  of 
flightdeck  observation  using  trained  observers.  This 
method  allowed  for  direct  observation  of  flight  crew 
procedures  and  recording  of  events  by  a  flightdeck 
observer.  The  second  method  entailed  indirect  obser¬ 
vation  and  included  the  use  of  structured  interviews 
and  a  questionnaire.  The  structured  interview  was 
completed  by  flight  crews  immediately  after  their 
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Figure  1.  Illustration  of  the  three  different  CDTI  Platforms  that  were  evaluated  in  OpEvai-2. 
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participation  in  each  flight  period.  The  questionnaire 
was  completed  immediately  after  flight  crews  partici¬ 
pated  in  their  final  OpEval-2  flight  period. 

Flightdeck  Observers  and  Training 

Nineteen  flightdeck  observers  participated  in 
OpEval-2;  all  had  previous  flightdeck  experience, 
either  as  flight  crews  or  as  observers.  Prior  to  OpEval- 
2,  the  observers  participated  in  a  training  program 
that  consisted  of  conference  calls,  email  distribution 
of  materials,  attendance  at  pre-OpEval-2  simulations, 
and  an  eight-hour,  on-site  orientation  session  the  day 
before  OpEval-2  began.  The  on-site  orientation  ses¬ 
sion  provided  a  review  of  CDTI  platform  operating 
procedures,  and  the  final  briefing  on  the  flight  sched¬ 
ule.  Observers  were  trained  in  the  following  areas: 

•  Overview  and  use  of  the  data-collection  materials 

•  Normal  flight  crew  operations  procedures,  including 
use  of  checklists,  typical  workload  management  pro¬ 
cedures,  and  sterile  cockpit  requirements 

•  Structured  interview  techniques 

•  OpEval-2  aircraft  and  flight  assignments 

•  Review  of  the  structure  and  timing  of  the  OpEval-2 
flight  periods  to  which  they  were  assigned. 

The  majority  of  flightdeck  observer  data  were  col¬ 
lected  by  direct  observation  of  flight  crew  procedures 
by  an  observer  occupying  the  jump  seat  of  the  trans¬ 
port  category  aircraft,  or  one  of  the  rear  seats  of  the 
single-pilot,  general  aviation  aircraft.  The  single  pilot 
aircraft  also  carried  a  safety  pilot  occupying  the  right 
seat.  Ail  flight  crews  were  instructed  to  perform  their 
normal  duties  without  regard  to  the  presence  of  the 
observer,  and  to  freely  exercise  their  judgment  to 
terminate  data  collection  if  in-flight  circumstances 
required  it.  Flight  crews  also  were  asked  to  verbalize 
their  thought  processes  so  that  the  observers  could 
gain  a  better  understanding  of  how  the  crews  used 
CDTI. 

Observers  were  instructed  not  to  interact  directly 
with  crewmembers,  except  as  required  to  collect  data 
for  the  ASSA  and  Initial  and  Final  Approach  Spacing 
applications.  In  the  instances  where  flight  crew-ob¬ 
server  interaction  was  necessary,  data  collection  oc¬ 
curred  only  during  ground  operations  with  the  aircraft 
stopped  and  parking  brakes  set.  Although  observers 
were  asked  to  make  all  other  observations  silently  as 
the  flight  crew  performed  their  normal  cockpit  duties, 
the  observer  was  considered  a  member  of  the  flight 
crew  in  all  other  respects  and  was  expected  to  call 
attention  to  any  safety  situation  requiring  the  atten¬ 
tion  of  the  flight  crew. 


Flightdeck  Observer  Forms 

Flightdeck  observers  recorded  their  observations 
on  the  following  forms,  which  were  designed  for 
directly  observing  OpEval-2  applications  and  proce¬ 
dures  during  flight.  A  brief  description  of  each  form 
is  given  in  the  following  section  and  the  actual  forms 
used  in  OpEval-2  are  included  in  Appendix  B: 

•  Checklists  Form 

•  Visual  Acquisition  and  Traffic  Awareness  Form 

•  Procedures  Forms 

•  Modified  Cooper-Harper  Workload  Scale 

•  Observers  also  administered  the  following  materials 
to  flight  crews  at  the  end  of  OpEval-2  flight  periods: 

•  Post-Flight  Structured  Interview 

•  Flight  Crew  Questionnaires 

Checklists  Form 

Th  is  Checklists  Form  was  used  to  determine  the 
effects,  if  any,  of  CDTI  platforms  on  flight  crew  use 
of  checklists.  Observers  recorded  whether  checklists 
were  completed,  not  completed,  suspended,  inter¬ 
rupted,  missed  by  the  crew,  etc.  A  frequency  analysis 
was  used  to  analyze  this  data  and  identify  the  most 
frequent  causes  of  checklist  intrusions.  Written  com¬ 
ments  from  the  forms  were  used  to  illustrate  examples 
of  specific  types  of  intrusions. 

Visual  Acquisition  and  Traffic  Awareness  Form 
This  form  was  used  to  record  how  flight  crews  used 
CDTI  to  aid  in  visual  acquisition  of  traffic  in  the 
Inner  and  Outer  Covey  patterns.  The  analysis  of  this 
data  focused  on  the  order  in  which  information  was 
accessed  to  support  target  acquisition  during  OpEval- 
2  flight  periods.  In  particular,  the  analysis  determined 
the  extent  to  which  the  introduction  of  CDTI  af¬ 
fected  the  procedure  typically  used  to  acquire  targets 
visually.  A  frequency  analysis  was  used  to  assess  the 
effects  of  displays  on  the  order  in  which  information 
was  accessed  to  support  target  acquisition.  Observers’ 
written  comments  were  used  to  supplement  the  fre¬ 
quency  analysis. 

Procedures  Forms 

Two  OpEval-2  Procedures  Forms  were  used  by 
observers:  the  CDTI  Build  2/4  Procedures  Form  and 
the  MX-20  Procedures  Form.  No  forms  were  devel¬ 
oped  for  the  Prototype  Avionics  platform  because  its 
development  was  in  an  early  stage.  Observers  used  the 
Procedures  Forms  to  assess  crew  use  of  recommended 
CDTI  techniques  as  defined  in  flight  crew  maneuver 
cards.  Analysis  of  data  from  the  forms  determined  the 
frequency  with  which  crews  used  procedures  defined 
in  the  maneuver  cards  or  deviated  from  these 
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procedures  and  created  their  own  procedures.  Devia¬ 
tions  from  suggested  procedures  may  indicate  flight 
crew  preferences  or  strategies  for  completing  a  proce¬ 
dure  and  these  may  be  more  efficient  or  “better”  than 
the  strategies  defined  in  the  maneuver  cards.  From  a 
certification  perspective,  such  preferences  or  strate¬ 
gies  may  represent  acceptable  and  safe  practices  for 
use  of  CDTI  for  OpEval-2  applications.  Written 
comments  from  this  form  were  used  to  describe  spe¬ 
cific  deviations  in  procedures  that  occurred  frequently. 

Modified  Cooper-Harper  Workload  Scale 
The  Modified  Cooper-Harper  Workload  Scale  was 
used  by  observers  to  assess  flight  crew  workload  of  the 
Pilot  Flying  (PF)  and  Pilot  Not  Flying  (PNF)  imme¬ 
diately  after  the  crew  used  CDTI  to  perform  the  final 
approach  spacing  application.  The  flight  crew  used  a 
scale  that  ranged  from  1  to  10  to  assign  a  workload 
rating.  Each  number  corresponded  with  a  qualitative 
description  of  the  workload  rating.  For  example,  a 
rating  of  1  corresponded  with  “Mental  effort  is  mini¬ 
mal  and  desired  CDTI  performance  is  easily  attain¬ 
able”  whereas  a  rating  of  10  corresponded  with 
“Acceptable  CDTI  performance  cannot  be  accom¬ 
plished  reliably”.  The  ratings  provided  by  flightcrew 
members  were  analyzed  for  possible  differences  due  to 
several  final  approach  spacing  factors  (e.g.,  time  of 
day,  spacing  interval). 

Post-Flight  Structured  Interview 
Following  each  OpEval-2  flight  period,  flightdeck 
observers  debriefed  the  participating  flight  crews  by 
conducting  a  structured  interview.  The  interview 
consisted  of  nine  (MX-20)  or  ten  (Build  2/4  and 
Prototype  Avionics)  questions  and  was  audiotaped. 
Interview  questions  focused  on  procedures  associated 
with  CDTI/MX-20  platform,  training  and  prepara¬ 
tion,  interaction  with  ATC,  and  anjr  tradeoffs  that 
may  have  been  associa  ted  with  the  use  of  CDTI  fo  r  the 
OpEval-2  applications.  Fight  crew  members,  were 
interviewed  together  to  promote  discussion  and  elicit 
additional  information.  After  each  interview,  observ¬ 
ers  created  a  written  summary  of  the  audiotaped 
recording  that  was  used  in  the  data  analysis.  The  flight 
crew  responses  to  the  questions  in  the  structured 
interview  form  were  categorized  and  aggregated  with 
the  flight  crew  responses  to  the  open-ended  questions 
from  the  respective  flight  crew  questionnaires  to  iden¬ 
tify  operational  and  human  factors  issues  associated 
with  use  of  CDTI  for  OpEval-2  applications. 


Flight  Crew  Questionnaires 

CDTI  Build  2/4  and  Prototype  Platforms 
A  173-item  questionnaire  was  the  primary  means 
for  gathering  information  on  flight  crew  demograph¬ 
ics,  previous  CDTI  experience  and  training,  and 
acceptability  of  the  following  characteristics  associ¬ 
ated  with  OpEval-2  applications: 

*  CDTI  functionality 

*  Use  of  color  in  display  modes 

*  Display  symbology 

*  Display  features 

*  Display  location  and  readability 

Descriptive  statistics  are  reported  for  the  question¬ 
naire  data,  and  factor  analyses  were  used  to  determine  if 
particular  CDTI  features  or  functional  components 
could  be  grouped  according  to  some  underlying  factor. 
Responses  to  open-ended  questions  were  analyzed  to 
identify  other  issues  that  the  flight  crew  judged  to  be 
pertinent.  Open-ended  responses  from  this  question¬ 
naire  were  aggregated  with  responses  from  the  structured 
interviews  to  summarize  operational  and  human  factors 
issues  associated  with  OpEval-2  applications, 

MX-20  Platform 

A  163-item  questionnaire  served  as  the  primary 
means  for  gathering  information  on  crew  demograph¬ 
ics,  previous  MX-20  experience  and  training,  and 
acceptability  of  the  following  characteristics  associ¬ 
ated  with  OpEval-2  applications: 

*  CDTI  functionality 

*  Use  of  color  in  display  modes 

*  Display  symbology 

*  Display  features 

*  Display  location  and  readability 

The  analysis  of  this  questionnaire  was  similar  to 
that  used  m  analyze  the  Build  274  questionnaire. 

Data  Analysis 

All  data  were  screened  for  errors,  outliers,  and  other 
anomalies  that  would  bias  statistical  analyses.  Descrip¬ 
tive  statistics  and  frequency  analyses  were  computed 
where  appropriate,  as  were  ANOVA  and  principal  com¬ 
ponents  analysis.  The  following  section  provides  a  sum¬ 
mary  of  results  that  are  based  on  analyses  of  data  from 
each  of  the  flightdeck  observer  forms.  OpEval-2  applica¬ 
tions  and  effects  associated  with  CDTI  platforms  also 
are  reported  in  this  section. 
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RESULTS 

Airport  Surface  Situation  Awareness 

A  set  of  circuitous  taxi  routes  was  designed  to  evaluate 
the  ability  of  flight  crews  to  accurately  navigate  with 
either  an  electronic  surface  map  or  a  paper  surface  map. 
The  routes  were  created  to  assess  the  extent  to  which  the 
CDTI  platform  could  be  used  for  the  ASSA  application 
to  improve  the  situation  awareness  of  flight  crews  during 
airport  surface  operations.  This  evaluation  assessed  flight 
crews’  ability  to  identify  specific  geographic  locations  on 
the  airport  surface  and  the  relative  location  of  other 
aircraft  and  surface  vehicles.  Routes  were  predefined, 
named,  and  provided  to  the  controllers  in  graphical  and 
text  format  and  to  the  flight  crews  in  written  text  format. 


Figure  2  shows  that  the  flightdeck  observers  noted 
that  53%  of  flight  crews  used  CDTI  to  identify 
Traffic  to  Follow  (TTF)  during  taxi  operations.  Of 
this  group,  a  larger  percentage  of  flight  crews  used  an 
electronic  surface  map  (65%  vs.  40%  with  a  paper 
surface  map).  Additional  analyses  revealed  that  the 
flightdeck  observers  reported  that  flight  crews  used 
CDTI  for  enhancing  surface  traffic  awareness  63%  of 
the  time.  Of  these  positive  responses,  82%  of  flight 
crews  used  an  electronic  surface  map,  and  40%  had  a 
paper  surface  map  (Figure  4).  Flightdeck  observers 
also  noted  that  53%  of  flight  crews  used  CDTI  to 
assess  the  location  of  traffic  on  the  airport  surface.  Of 
this  group,  76%  used  an  electronic  surface  map,  and 
27%  used  a  paper  surface  map  (Figure  4). 


■Surface  Map  (n=  17)  DNo  Surface  Map  (n^  15) 


Used  CDTI  to  Locate  Traffic  to  Follow 


Figure  2.  Use  of  CDTI  to  locate  Traffic-to-Follow  (TTF)  in  taxi  sequence  by  flight  crews 
with  an  electronic  surface  map  and  those  with  a  paper  surface  map. 


■  Surface  Map  (n=17)  DNo  Surface  Map  (n-15) 


Yes  No  Missing  Data 


Used  CDTI  for  Surface  Traffic  Awareness 


Figure  3.  Use  of  CDTI  for  surface  traffic  awareness  by  flight  crews  with  an  electronic 
surface  map  and  those  with  a  paper  surface  map. 
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Used  CDTI  to  Locate  Traffic  on  Surface 


Figure  4.  Use  of  CDTI  to  assess  traffic  location  on  the  airport  surface  by  flight  crews 
with  an  electronic  surface  map  and  those  with  a  paper  surface  map. 


Flightdeck  observers  also  recorded  flight  crew  use 
of  two  CDTI  platform  features;  the  Target  Selection 
function  and  the  Ground  Track  Vector  function:. 
Overall,  3 1  %  of  flight  crews  used  the  Target  Selection 
function  on  CDTI  to  Identify  TTF  during  taxi  opera¬ 
tions.  Approximately  12%  of  flight  crews  used  the 
Ground  Track  Vector  function  on  CDTI  to  evaluate 
the  target  speed  during  airport  surface  operations. 
These  percentages  were  nearly  equivalent  for  flight 
crews  with  an  electronic  surface  map  and  those  with 
a  paper  surface  map. 

Flightdeck  observer  data  indicated  that  one-third 
of  the  flight  crews  utilized  CDTI  to  identify  taxiway 
locations  on  the  airport  surface.  Of  this  group,  65% 
of  flight  crews  with  an  electronic  surface  map  used  the 
CDTI  to  identify  taxiway  locations  during  surface 
navigation.  None  of  the  flight  crews  with  a  paper 
surface  map  used  CDTI  for  this  purpose. 

Flightdeck  observers  also  recorded  whether  flight 
crews  correctly  identified  the:  most  proximal  surface 
traffic  where  they  were  asked  this  question..  The  data 
indicated  that  only  12  of  the  32  flight  crews  re¬ 
sponded  to  this  question.  Of  the  1 2  flight  crews,  7  had 
the  electronic  surface  map,  and  all  correctly  identified 
the  most  proximal  traffic.  Four  of  the  five  flight  crews 
with  a  paper  surface  map  correctly  identified  the  most 
proximal  traffic. 

FAROA 

Flight  crews  were  instructed  to  utilize  the  CDTI  to 
assess  runway  occupancy  prior  to  entering  or  crossing 
runways  (FAROA  ground).  During  airborne  opera¬ 
tions  from  the  final  approach  fix  through  rollout, 
flight  crews  were  instructed  to  utilize  CDTI  to 


continuously  assess  runway  occupancy  (FAROA  air). 
Flightdeck  observers  recorded  data  for  the  FAROA 
application  during  these  airborne  and  ground  events. 
Addfeionally,  during  Flight  Periods  1  and  2,  a  surface 
vehicle  and  an  aircraft  were  positioned  at  two  loca¬ 
tions  on  the  active  runway  to  evaluate  the  effect  of  the 
CDTI  on  flight  crew  awareness  of  runway  occupancy. 

FAROA  Air 

Flightdeck  observer  data  indicated  that  during 
FAROA  Air  operations  62%  of  flight  crews  used  the 
CDTI  to  evaluate  the  runway  environment  for  pos¬ 
sible  conflicts.  Figure  5  shows  that  within  this  group, 
71%  of  flight  crews  used  an  electronic  surface  map, 
and  56%  used  a  paper  surface  map.  During  final  on 
low  approaches,  flightdeck  observers  reported  that 
flight  crews  using  CDTI  equipped  with  an  electronic 
surface  map  were  more  effective  at  determining  when 
the  runway  was  occupied  than  flight  crews  with  the 
paper  surface  map.  Eight  flight  crews  with  an  elec¬ 
tronic  surface  map  identified  the  traffic  on  the  run¬ 
way,  whereas  flight  crews  with  a  paper  surface  map 
failed  to  identify  the  traffic  on  the  runway.  These 
results  suggest  that  the  addition  of  an  electronic  surface 
map  may  aid  in  identifying  traffic  on  a  runway. 

Flightdeck  observers  also  recorded  CDTI  display 
modes  that  were  used  most  frequently  by  flight  crews 
during  the  FAROA  Air  application.  Observer  data  indi¬ 
cated  Arc  mode  was  used  51%  of  the  time  by  the  flight 
crews  who  had  a  paper  surface  map.  The  Compass  Rose 
and  No  Arc/No  Rose  modes  both  were  used  nearly  21% 
of  the  time.  The  Compass  Rose  mode  was  selected  39% 
of  the  time  and  the  Arc  mode  14%  of  the  time  by  the 
flight  crews  who  had  an  electronic  surface  map. 
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■  Surface  Map  (n-49)  DNo  Surface  Map  (n=67) 


100 


Yes  No  Missing  Data 

Used  CDTI  to  Evaluate  Runway  Environment 


Figure  5.  Use  of  the  CDTI  during  FAROA  Air  operations  to  evaluate  runway  environment  for 
possible  conflicts  by  flight  crews  with  an  electronic  surface  map  and  those  with  a  paper 
surface  map. 

■  Surface  Map  (n=17)  □  No  Surface  Map  (n=15) 


Used  CDTI  for  Runway  Occupancy  Awareness 


Figure  6.  Use  of  the  CDTI  during  FARAO  Ground  operations  for  runway  occupancy  awareness 
by  flight  crews  with  an  electronic  surface  map  and  those  with  a  paper  surface  map. 


Flight  crew  use  of  two  CDTI  platform  features,  the 
Range  function  and  the  Ground  Track  Vector  func¬ 
tion,  was  recorded  by  flightdeck  observers  during  the 
FAROA  Air  application.  Approximately  53%  of  the 
flight  crews  used  the  Range  function  to  adjust  viewing 
range  while  on  the  final  approach.  Of  this  group,  69% 
of  flight  crews  with  an  electronic  surface  map  utilized 
the  Range  function,  and  41%  of  flight  crews  with  a 
paper  surface  map  used  the  function.  Observer  data 
indicated  that  only  23%  of  flight  crews  used  the 
Ground  Track  Vector  function  to  evaluate  the  target 
speed  of  the  leading  aircraft  or  to  identify  an  aircraft 
accelerating  onto  the  runway.  These  flight  crews  were 
split  evenly  between  those  with  an  electronic  surface 
map  and  those  with  a  paper  surface  map. 


FAROA  Ground 

Flightdeck  observers  reported  that  during  FAROA 
Ground  operations,  56%  of  flight  crews  used  CDTI 
to  assess  runway  occupancy.  Figure  6  reveals  that 
nearly  71%  of  flight  crews  with  an  electronic  surface 
map  and  40%  of  crews  with  a  paper  surface  map  used 
CDTI  to  support  this  task.  A  majority  of  flight  crews 
used  CDTI  to  check  for  traffic  on  the  approach,  and 
half  of  the  crews  used  the  CDTI  to  check  the  departure 
corridors  for  possible  conflicts.  Again,  flight  crews 
with  an  electronic  surface  map  utilized  CDTI  with 
greater  frequency  (71%  for  approach  and  59%  for 
departure,  respectively),  than  flight  crews  with  a  pa¬ 
per  surface  map  (47%  for  approach  and  40%  depar¬ 
ture,  respectively). 
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Flightdeck  observers  noted  that  65%  of  flight  crews 
used  the  Range  function.  Flight  crews  with  a  surface 
map  tended  to  use  the  Range  function  more  often 
(71%)  than  those  with  no  surface  map  (53%).  Only 
12%  of  the  crews  with  a  surface  map  and  none  of  the 
crews  without  a  surface  map  used  the  Ground  Track 
Vector  function  to  evaluate  target  speed. 

Checklists 

Flightdeck  observers  recorded  flight  crew  behavior 
associated  with  checklists  used  during  the  flight.  Data 
from  the  Checklist  Observer  Form  were  used  to  deter¬ 
mine  the  effects  of  the  four  display  types  on  flight 
crew  use  of  checklists.  Observers  recorded  whether 
checklists  were  Completed,  Suspended,  Interrupted, 
Not  Completed,  Missed  by  the  Crew,  or  Missed  by 
the  Observer.  In  addition,  observers  were  told  to  use 
Other  as  a  category,  if  necessary.  Preliminary  analyses 
showed  very  low  counts  in  the  Suspended,  Inter¬ 
rupted,  Not  Completed,  Missed  by  the  Crew,  and 
Missed  by  the  Observer  categories.  These  categories 
were  later  pooled  into  two  more  general  categories; 
Not  Completed,  which  included  the  original  Not 
Completed  and  Missed  by  Crew- categories;  and.  Com¬ 
pleted,  which  the  included  the  original  Completed, 
Suspended,  and  Interrupted  categories. 

A  frequency  analysis  was  used  to  determine  check¬ 
list  completion  rates  across  the  four  different  traffic 
display  types.  After  recoding  the  data  into  the  Com¬ 
pleted  and  Not  Completed  categories,  the  data  were 
statistically  analyzed  using  a  chi-square  test.  The 
results  of  the  frequency  analysis  are  shown  in  Figure 


7,  which  shows  checklist  completion  rates  by  display 
type.  Overall,  the  checklist  completion  rate  was  equiva¬ 
lent  across  display  types,  and  it  was  greater  than  98% 
for  all  display  types.  Completion  rates  were  not  af¬ 
fected  by  time  of  day  (i.e.,  day  vs.  night)  in  which  the 
OpEval-2  flight  periods  occurred. 

Although  use  of  CDTI  did  not  appear  to  affect 
checklist  completion  rates,  written  comments  by  some 
of  the  flight  crews  using  each  display  type  suggested 
that  the  timely  completion  of  checklists  came  with  a 
cost  (viz.,  additional  workload).  For  example,  inter¬ 
pretation  of  the  flight  crew  data  suggested  that  two  of 
the  five  checklist  suspensions  and  five  of  the  ten 
checklist  interruptions  were  due  to  CDTI.  Another 
factor  that  may  have  induced  workload  was  the  closed 
traffic  pattern  operations  used  in  OpEval-2,  which 
required  repetitive  checklist  operations  that  are  not 
typical  in  normal  flight  operations,  thereby  increasing 
workload.  The  checklist  data  do  mm  reveal  whether 
the  Increase  In  perceived  workload  was  due  to  the 
novel  displays,  per  se*  the-  higher- than-normal  check¬ 
list  repetition  resulting  from  the  dosed  traffic  pat¬ 
terns  used  in  OpEval-2,  or  the  interaction  of  these  two 
factors.  Some  flight  crews  reported  that  their  per¬ 
ceived  workload  was  reduced  as  they  became  more 
familiar  with  the  displays. 

Visual  Acquisition  and  Traffic  Awareness  Form 

Observers  used  the  Visual  Acquisition  and  Traffic 
Awareness  Form  to  record  the  order  and  manner  in 
which  the  flight  crews  used  three  different  information 
methods  to  enhance  traffic  awareness  while  in  the  Outer 
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Figure  7.  Checklist  completion  rate  by  platform  type. 
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and  Inner  Covey  traffic  patterns.  These  three  methods 
included  use  of  the  CDTI,  ATC  traffic  advisories,  and 
visual  acquisition  of  traffic. 

The  analysis  of  the  acquisition  data  focused  on  the 
order  in  which  information  was  accessed  to  support 
target  acquisition  and  traffic  awareness  during  OpEval- 
2  flight  periods.  In  particular,  the  analysis  sought  to 
determine  the  extent  to  which  the  introduction  of  the 
display  platform  affected  the  procedure  typically  used  to 
acquire  targets  (i.e.,  ATC  advisory/visual  acquisition), 
during  both  the  night  and  the  day  flight  periods.  A 
frequency  analysis,  which  included  a  chi-square  test,  was 
used  to  determine  if  the  three  methods  were  used  equally 
often.  The  frequency  analysis  was  also  used  to  define  the 
order  in  which  methods  were  used  to  support  target 
acquisition  and  traffic  awareness. 

The  chi-square  test  for  the  first  method  used  (CDTI) 
was  significant,  %2(2)= 17.49,  pc.0001.  During  day  and 
night  events,  the  observed  usage  rate  for  the  CDTI 
method  was  significantly  higher  than  expected,  whereas 
the  observed  usage  for  the  visual  method  was  signifi¬ 
cantly  lower  than  expected.  As  Figures  8  and  9  show, 
flight  crews  initially  identified  nearly  two-thirds  of  the 
traffic  targets  using  the  traffic  display.  The  chi-square 
test  for  the  second  (AT C  T raffic  Advisors) ,  %2  (2)  =  1 0 . 94 , 
pc. 004,  and  third  methods  (Visual  Acquisition),  X2 
(2)  =  10.94,  pc. 004,  used  also  were  significant. 
Interestingly,  Figure  8  reveals  that  flight  crews  were 
somewhat  more  likely  to  use  ATC  traffic  advisories  than 
visual  sighting  as  the  second  method  during  day  opera¬ 
tions.  However,  during  night  operations  the  flight  crews 
were  far  more  likely  to  use  visual  sighting  than  ATC 


traffic  advisories  as  the  second  method.  This  effect  is 
attributable  to  the  better  visibility  that  prevailed  during 
OpEval-2  night  flight  periods.  The  traffic  awareness 
benefits  associated  with  the  CDTI  were  more  apparent 
during  day  flight  periods  when  the  visibility  was  poor 
(hazy  VFR).  Overall,  about  75%  of  all  traffic  events 
involved  use  of  the  CDTI. 

Modified  Cooper-Harper  Workload  Scale 

Flightdeck  observers  used  the  Modified  Cooper- 
Harper  Workload  Scale  to  obtain  a  workload  rating 
from  the  Pilot  Flying  (PF)  immediately  after  the  flight 
crew  used  the  platform  to  perform  the  Final  Approach 
Spacing  application.  After  each  full-stop  landing,  flight 
crews  using  CDTI  Build  2/4,  Advanced  Prototype,  and 
Basic  Prototype  platforms  rated  perceived  workload 
using  the  scale  items,  which  were  read  by  the  observer 
after  the  flight  crew  set  the  aircraft  brakes.  Because  flight 
crew  tasks  were  not  identical  and  comparisons  between 
platform  types  based  on  workload  ratings  alone  would  be 
misleading,  the  workload  ratings  were  analyzed  within 
platform  types  for  possible  differences  due  to  time  of 
day  (i.e.,  day  vs.  night)  and  final  approach  spacing 
interval  (closing  to  5  nm  vs.  3  nm).  No  significant 
differences  were  found  for  time  of  day  or  final  ap¬ 
proach  spacing  interval. 

Pilot  workload  ratings  for  the  Final  Approach 
Spacing  application  varied  by  CDTI  platform.  Pilot 
workload  ratings  are  shown  in  Table  2.  The  rating  2.9 
corresponds  to  the  Modified  Cooper-Harper  scale 
value  “Mental  effort  is  required  to  attain  acceptable 
performance.”  The  rating  8.2  corresponds  to  the 
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Figure  8.  Three  methods  used  for  detecting  traffic  and  the  order  in  which  flight 
crews  used  these  methods  during  OpEval-2  day  flight  periods. 
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Figure  9.  Three  methods  used  for  detecting  traffic  and  the  order  fn  which  flight 
crews  used  these  methods  during  OpEvaf-2  night  flight  period's. 


Table  2.  Mean  pilot  workload  ratings  as  a  function  of  CDTI  platform. 


Platform  Type 

N 

Mean  Rating 

Standard  Deviation 

CDTI  Build  2/4 

42 

2.9 

2.2 

Basic  Prototype 

21 

8.2 

1.5 

Advanced  Prototype 

15 

4.8 

0.9 

Modified  Cooper-Harper  scale  value  “Maximum 
mental  effort  is  required  to  avoid  large  or  numerous 
errors/5  The  rating  4.8  corresponds  to  the  Modified 
Cooper-Harper  scale  value  “  High  mental  effort  is 
required  to  attain  acceptable  performance/5 

Flight  crew  tasks  for  the  Final  Approach  Spacing 
application  varied  as  a  function  of  platform  type.  Flight 
crews  using  CDTI  Build  2/4  and  the  Basic  Prototype 
platforms  were  to  achieve  a  desired  spacing  at  the  thresh¬ 
old  by  observing  the  raw  speed,  closure  and  range  data  on 
CDTI  and  determining  the  necessary  speed  adjustments 
to  achieve  the  desired  spacing  behind  TTF.  Flight  crews 
using  the  Advanced  Prototype  were  to  adjust  speed  in 
response  to  a  speed  command,  which  was  computed  as 
a  function  of  planned  approach  speeds,  current  speeds 
and  distance  to  threshold.  This  speed  command  was 
displayed  as  a  supplementary  cursor  on  the  EFIS  air¬ 
speed  display.  For  these  flight  crews,  matching  current 
speed  with  commanded  speed  should  have  resulted  in 
the  targets  spacing  in  seconds  behind  TTF.  Flight  crews 
using  MX-20  did  not  perform  spacing  tasks. 


None  of  the  displays  had  been  certified  for  use  in 
the  Final  Approach  Spacing  application  and  only  the 
Advanced  Prototype  platform  had  design  features 
that  specifically  supported  the  spacing-at-threshold 
task.  Nevertheless,  flight  crews  using  CDTI  Build  2/ 
4  platform  rated  the  workload  lower  than  the  flight 
crews  using  the  other  two  platform  types.  Flight  crews 
using  the  Basic  Prototype  rated  the  workload  highest, 
and  flight  crews  using  the  Advanced  Prototype  dis¬ 
play  rated  workload  as  intermediate  (i.e.,  lower  than 
Basic  Prototype  platform,  but  higher  than  CDTI 
Build  2/4  platform). 

Anecdotal  evidence  from  the  observers  suggests 
that  in  some  cases,  the  PF  may  have  rated  both  the 
Initial  Approach  Spacing  and  Final  Approach  Spac¬ 
ing  tasks  and  not  just  the  latter  task.  Workload  ratings 
should  be  interpreted  with  caution  due  to  the  differ¬ 
ences  between  crew  tasks,  platform  capability  and 
small  sample  sizes  for  some  ratings. 
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Procedures  Form 

CDTI  Build  2/4  Platform 

The  CDTI  Build  2/4  platform  flight  crew  Mission 
Cards  specified  tasks  to  be  accomplished  using  CDTI 
information.  For  the  Departure  Spacing  application, 
the  task  was  to  determine  the  assigned  separation 
interval  (either  4.5  nm  or  6.0  nm)  between  ownship 
and  TTF  on  departure,  and  then  release  brakes  for 
takeoff.  A  completion  standard  was  not  specified  in 
the  Mission  Card  for  this  task,  and  flightdeck  observ¬ 
ers  used  their  own  judgment  to  determine  whether  the 
task  was  completed. 

The  Initial  Approach  Spacing  application  task  re¬ 
quired  flight  crews  to  establish  and  maintain  the 
target  separation  interval  on  the  downwind  leg  as 
defined  for  that  flight  period.  The  target  value  was 
either  7  nm  or  9  nm,  with  a  completion  standard  of  ± 
1  nm.  The  Final  Approach  Spacing  application  task 
required  flight  crews  to  close  to  the  target  separation 
distance  at  the  runway  threshold.  Target  values  were 
either  5  nm  or  3  nm,  with  a  completion  standard  of 


+  1.0  nm  to  0.5  nm.  Task  completion  rates  for  flight 
crews  using  the  CDTI  Build  2/4  platform  are  pre¬ 
sented  in  Table  3. 

Completion  rates  were  approximately  the  same  for 
the  three  OpEval-2  applications.  These  data  provide 
some  insight  into  the  ability  of  flight  crews  to  follow 
a  procedure  for  achieving  an  assigned  spacing  based 
on  the  use  of  CDTI.  The  results  may  be  useful  for 
discussing  minimum  acceptable  reliability  of  perfor¬ 
mance  in  an  actual  operational  setting  when  ATC  may 
be  relying  on  a  flight  crew's  ability  to  achieve  an 
assigned  spacing. 

The  flight  crew  Mission  Cards  also  provided  flight 
crews  using  the  CDTI  Build  2/4  platform  with  sug¬ 
gested  techniques  for  completing  the  applications. 
Techniques  that  related  specifically  to  the  use  of 
CDTI  information  were  evaluated  by  the  flightdeck 
observers  in  accordance  with  Table  4.  These  tech¬ 
niques  were  included  in  the  Procedures  Form,  which 
was  used  to  record  the  data  for  each  circuit  flown  by 
aircraft  in  the  Outer  Covey  pattern.  These  data  were 


Table  3.  CDTI  Build  2/4  platform  task  completion  by  OpEval-2  application  (in  percent). 


Task  Completed 

Departure 

Spacing 

Initial  Approach 
Spacing 

Final  Approach 
Spacing 

Yes 

83.3 

86.7 

88.5 

No 

10.0 

8.9 

11.5 

Missing  Data 

6.7 

4.4 

0.0 

Table  4.  CDTI  Build  2/4  platform  tasks  and  recommended  techniques  for  Departure  Spacing,  Initial 
Approach  Spacing  and  Final  Approach  Spacing  applications. 


Recommended  Techniques  for  OpEval-2 
Applications 

Departure 

Spacing 

Initial 

Approach 

Spacing 

Final  Approach 
Spacing 

Determine 

Establish  and 

Establish  and 

departure 

maintain  desired 

maintain  desired 

separation 

initial  separation 

final  approach 

interval 

distance 

separation 

Set  range  ring  to  separation  distance  (SET_RR) 

Adjust  display  range  prior  to  takeoff 
(RUP.RDN) 

Set  the  desired  altitude  range  for  traffic  display 
for  the  application  (USE__LVL) 

Use  vector  display  to  evaluate  target  speed 
(USEJVEC) 

Select  TGT  (SEL)  to  identify  target  (SEL_TGT) 

Set  vector  time  to  2  min  Long  or  1.5  min  Short 
(SETJVEC) 

Eval  spacing  by  PNF  only  / 


/  /  / 

/ 

/  / 

/  / 

/  / 

/  /  / 
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aggregated  across  circuits  to  compute  overall  percent¬ 
age  of  use.  Percentages  for  each  technique  are  pre¬ 
sented  by  OpEval-2  application. 

Departure  Spacing 

Data  on  the  use  of  each  of  the  techniques  during 
the  Departure  Spacing  application  are  provided  in 
Figure  10,  The  most  notable  deviation  from  the 
recommended  techniques  was  in  the  use  of  the  alti¬ 
tude  range  filter  (USE_LVL).  This  display  feature, 
which  can  be  adjusted  by  the  flight  crew,  determines 
the  altitude  range  in  which  ADS-B  targets  would  be 
displayed.  The  default  value  displays  targets  up  to 
2500  feet  above  and  below  ownship.  In  LK-mode, 
which  was  the  recommended  departure  setting,  the 
range  above  ownship  changes  to  9500  feet  (the  system 
default  value),  or  a  value  set  by  the  flight  crew.  The 
purpose  of  the  recommendation  is  to  ensure  that  the 
display  continuously  shows  the  target  of  interest  dur¬ 
ing  its  climb-out  on  departure.  Most  flight  crews  did 
not  change  the  setting  prior  to  departure.  The  flight 
crew  Mission  Card  for  the  preceding  application  (Le., 
FAROA  Ground)  suggested  that  this  mode  be  se¬ 
lected;  thus,  further  adjustment  was  not  necessary.  In 
addition,  flightdeck  observers  reported  that  the  climb 
performance  of  departing  aircraft  during  OpEval-2 
was  such  that  the  aircraft  did  not  climb  above  the 
altitude  filter  setting  (and  thus  disappear  from  the 
display)  regardless  of  its  setting.  Therefore,  even  when 
they  did  not  set  CDTI  display  features  as  suggested, 
flight  crews  were  not  prompted  to  adjust  settings  to 
the  recommended  configuration  because  the  depart¬ 
ing  aircraft  did  not  disappear  from  the  display.  Fi¬ 
nally,  observers  noted  that  in  most  cases,  settings 
made  at  the  ramp  before  departure  were  often  not 
changed  for  the  duration  of  the  flight. 


In  most  aircraft,  the  Range  Ring  (SET_RR)  was  set 
to  a  value  appropriate  for  tasks  in  the  Departure 
Spacing  application.  Fewer  flight  crews  were  seen 
adjusting  or  heard  verbally  verifying  that  the  display 
range  was  set  prior  to  departure  (RUP_RDN) .  For  the 
short-spacing  departure  interval  (closing  to  4,5  nm), 
the  best  display  resolution  was  at  the  five-mile  scale; 
however,  a  longer  scale  also  could  be  used,  and  some 
flight  crews  may  have  decided  that  an  adjustment  was 
unnecessary  under  the  circumstances.  In  fact,  there 
would  have  been  no  loss  of  accuracy  from  using  the 
longer  display  range  if  flight  crews  had  highlighted 
the  target  using  the  SEL  function.  Target  Selection 
behavior  for  tasks  in  the  Departure  Spacing  applica¬ 
tion  was  not  assessed  specifically,  nor  did  observers 
record  their  own  observations  regarding  Target  Selec¬ 
tion.  However,  informal  discussions  with  observers 
indicate  that  many  flight  crews  retained  the  Target 
Selection  that  they  used  for  tasks  in  the  ASSA  and 
FAROA  Ground  applications,  which  preceded  De¬ 
parture  Spacing  and  occurred  during  taxi  operations. 
Once  selected,  the  range  to  the  target  to  the  nearest 
tenth  of  a  mile  would  be  available  in  its  displayed  data 
tag  regardless  of  display  range  setting  or  range  ring 
value.  In  lieu  of  the  range  ring,  this  information  could 
be  used  readily  to  support  tasks  in  the  Departure 
Spacing  application.  Direct  alphanumeric  readout  of 
target  range  using  the  Select  function  would  provide 
the  most  accurate  data  given  the  size  and  resolution  of 
the  display,  and  the  size  of  the  traffic  symbology. 

Similarly,  although  most  flight  crews  adjusted  vec¬ 
tor  display  time  to  the  recommended  value 
(SETJVTC),  some  did  not.  The  recommendation  to 
set  the  vector  time  supported  the  Initial  Approach 
Spacing  task,  and  not  the  immediate  tasks  In  the 
Departure  Spacing  application.  Flight  crews  who 
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Figure  10.  Flight  crew  techniques  using  CDTI  Build2/4  platform  for  the 
Departure  Spacing  application. 
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elected  hot  to  set  the  display  may  have  correctly 
determined  that  it  did  not  provide  useful  information 
for  the  tasks  in  the  Departure  Spacing  application. 
The  vector  display  is  time-based,  providing  an  indica¬ 
tion  of  ownship  position  at  a  selectable  future  time, 
and  the  tasks  in  the  Departure  Spacing  application 
were  distance-based.  This  particular  technique  also 
had  the  highest  missed  data  rate;  some  observers  notes 
recorded  difficulty  in  determining  the  status  of  the 
display  from  their  vantage  point  in  the  cockpit 
jumpseat. 

Initial  Approach  Spacing 

Data  on  the  use  of  techniques  during  the  Initial 
Approach  Spacing  application  appear  in  Figure  IT 
The  notable  deviations  from  the  recommended  tech¬ 
niques  during  tasks  associated  with  the  Initial  Ap¬ 
proach  Spacing  application  were  in  the  use  of  the 
Altitude  Filter  (SET_TGT)  and  use  of  the  Vector 
Time  Display  (USE_VEC)  to  evaluate  target  speed. 
The  lack  of  observed  compliance  with  these  tech¬ 
niques  is  due  in  part  to  a  difference  between  the 
techniques  suggested  on  the  Mission  Card  and  the 
techniques  listed  on  the  Procedures  Form.  Adjust¬ 
ment  of  the  altitude  filter  and  use  of  the  vector  display 
to  evaluate  target  speed  are  not  among  the  techniques 
recommended  on  the  Mission  Card.  Therefore,  it  is 
not  surprising  that  observed  compliance  is  low. 
Flightdeck  observer  comments  indicated  that  the  alti¬ 
tude  filter  setting  made  on  the  ramp  was  sometimes 
not  changed  for  the  duration  of  the  flight.  The  rela¬ 
tively  higher  compliance  with  the  SEL_TGT  tech¬ 
nique  and  several  recorded  observer  comments 
indicated  that  TTF  speed  information  was  most  com¬ 
monly  acquired  from  the  alphanumeric  readout  of 


selected  target  speed.  In  addition,  there  were  two 
recorded  observations  of  active  use  of  the  closure  rate 
display  to  capture  the  initial  approach  spacing  interval. 

The  highest  compliance  rate  for  techniques  in  the 
Initial  Approach  Spacing  application  was  setting  the 
vector  time  (SET  VEC)  to  recommended  values. 
However,  several  observer  comments  indicated  that 
high  compliance  rates  were  not  due  to  active  choices 
made  by  the  flight  crews.  Instead,  the  rates  were 
attributable  to  initial  selections  made  by  flight  crews 
while  they  were  on  the  ramp  or  during  previous 
application  tasks.  No  observer  reported  active  use  of 
the  vector  display  for  this  application.  As  was  the  case 
for  the  Departure  Spacing  application,  the  Initial 
Approach  Spacing  application  was  fundamentally  a 
distance-based,  in-trail  separation  task.  In  contrast, 
the  ownship  vector  is  time-based,  and  thus  would 
represent  the  correct  spacing  only  at  one  particular 
true  airspeed. 

Final  Approach  Spacing 

For  the  Final  Approach  Spacing  Application,  the 
Build  2/4  flight  crew  Mission  Card  specified  two  of 
the  five  techniques  evaluated  by  flightdeck  observers: 
use  ofTarget  Selection  (SEL_TGT)  and  evaluation  of 
spacing  by  the  PNF  only.  The  latter  technique  is  not 
a  CDTI  function;  rather,  it  is  a  crew  resource  manage¬ 
ment  technique  for  using  CDTI.  As  shown  in  Figure 
12,  Target  Selection  was  used  on  most  of  the  arrivals, 
whereas  evaluation  ofspacingby  PNF  only  was  judged 
by  observers  to  have  occurred  just  over  half  the  time. 
The  Target  Select  function  provides  essential  infor¬ 
mation  for  tasks  associated  with  the  Final  Approach 
Spacing  application,  including  the  ground  speed  of 
the  selected  target  and  closure  rate  display.  The  high 
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Figure  11.  Flight  crew  techniques  using  CDTI  Build2/4  platform  for  the  Departure 
Spacing  application. 
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Figure  12.  Flight  crew  techniques  using  CDTI  BuiId2/4  platform  for  the  Final 
Approach  Spacing  application. 


percentage  of  use  of  this  feature  may  reflect  the  impor¬ 
tance  of  such  information.  Flight  crews  reported 
using  this  information  as  a  primary  source  of  informa¬ 
tion  about  the  TTF  during  the  Final  Approach  Spac¬ 
ing  applications  a  finding  confirmed  by  the  flightdeck 
observers. 

The  lower  rate  associated  with  the  evaluation  of 
spacing  by  PNF  is  noteworthy?  given  that  the  intent  of 
the  technique  was  to  enable  the  PF  to  concentrate  on 
the  aircraft  control  tasks  associated  with  the  final 
approach  phase  of  flight.  When  using  this  technique, 
the  PNF  manipulates  the  traffic  display  and  provides 
the  PF  with  appropriate  information.  Flightdeck  ob¬ 
server  data  indicated  that  about  a  third  of  PFs  devoted 
some  attention  to  the  display  during  the  final  ap¬ 
proach  spacing  task.  One  explanation  for  these  results 
may  be  that  the  novelty  of  CDTI  captured  the  atten¬ 
tion  of  the  entire  flight  crew.  Prior  experience  with  the 
implementation  of  the  Traffic  Alert  and  Collision 
Avoidance  System  (TCAS)  suggests  that  this  novelty 
phenomenon  may  be  short-lived,  diminishing  with 
additional  experience.  More  effective  flightdeck  pro¬ 
cedures  based  on  a  thorough  task  analysis  of  the  target 
procedures  and  emphasis  during  training  also  may 
enhance  performance, 

Post-Flight  Structured  Interview  Form 

Flightdeck  observers  debriefed  flight  crews  using  a 
structured  interview.  A  content  analysis  of  the  result¬ 
ing  interview  summaries  was  conducted  to  identify 
the  operational  and  human  factors  issues  that 
crewmembers  found  to  be  particularly  relevant  to  the 
OpEval-2  applications.  Content  analyses  were  per¬ 
formed  on  35  interview  summaries  that  represented 
the  responses  of  38  unique  crewmembers.  Approxi¬ 
mately  430  individual  statements  were  evaluated  and 


categorized.  Crewmember  responses  for  all  CDTI 
platforms  were  represented.  Flight  crews  using  the 
CDTI  Build  2/4  platform  composed  nearly  53%  of 
the  responses. 

Interpretation  of  the  interview  data  was  compli¬ 
cated  by  factors  related  to  the  method  used  to  collect 
it.  For  example,  interviewers  were  free  to  choose  their 
own  method  of  transcription  and  subsequent  report¬ 
ing  of  the  data.  There  was  significant  variation  in  the 
level  of  detail  provided  for  analysis,  from  very  short, 
written  summaries  to  a  verbatim  transcript.  In  at  least 
one  case,  the  interview  questions  appeared  to  have 
been  answered  in  writing,  with  no  additional  discus¬ 
sion  by  the  interviewer  and  crewmembers.  Given 
these  factors,  it  is  possible  that  additional  issues  may 
have  been  raised  by  the  crew  and  not  captured  in  the 
written  summaries.  Furthermore,  the  relative  propor¬ 
tion  of  responses  within  the  issue  categories  shown  in 
Figure  13  may  not  be  reliable  because  these  propor¬ 
tions  are  based  on  a  small  number  of  crewmember 
responses.  Based  on  these  factors,  the  structured  in¬ 
terview  data  are  reported  in  aggregate,  with  the  intent 
of  providing  a  broad  overview  of  the  opinions  and 
concerns  about  operational  and  human  factors  issues 
that  were  expressed  by  flight  crews  who  participated 
in  OpEval-2. 

*  The  content  analysis  suggested  five  broad  categories 
of  issues: 

*  Applications  -  Direct  comments  on  OpEval-2  appli¬ 
cations. 

*  ATC  Interaction  -  Reported  effects  of  CDTI  on  the 
ATC  procedures,  communications,  or  comments  re¬ 
lated  to  flightdeck/ATC  procedures  integration. 

*  Crew  Resource  Management  (CRM)  -  Flight  crew 
interaction,  use  of  checklists,  flightdeck  procedures 
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design,  workload,  tradeoffs,  and  other  aspects  of 
flightdeck  operations. 

#  System  -  Comments  related  to  the  design  and  opera¬ 
tion  of  the  CDTI,  or  its  operating  context;  includes, 
controls,  displays,  installation  and  other  aspects  of 
usability. 

•  Training  -  Platform-specific  training  concerns  and 
OpEval-2  learning  effects. 


Applications 

Training  7o/o 


(Number  of  statements  is  in  parentheses) 

Figure  13.  Percentage  of  interview  statements  by 
category  for  ail  CDTI  platforms. 


The  two  largest  categories  are  System  and  Crew 
Resource  Management  (CRM),  followed  by  Train¬ 
ing,  ATC  Interaction,  and  Applications.  Positive  and 
negative  reactions  were  recorded  within  each  cat¬ 
egory,  but  the  general  trend  was  to  focus  on  system 
deficiencies  or  issues  that  crews  felt  needed  improve¬ 
ment.  The  responses  within  a  category  were  separated 
by  CDTI  platform  type  to  determine  if  the  platform 
types  were  associated  more  frequently  with  particular 
comments. 

System 

Mixed  Equipage 

Flight  crews  expressed  concerns  about  the  utility  of 
CDTI  platforms  in  a  mixed  or  partial  equipage  envi¬ 
ronment.  Two  ADS-B  links  were  evaluated  during 
OpEval-2,  and  each  was  “blind”  to  the  other.  That  is, 
flight  crews  who  used  a  CDTI  platform  with  one  of 
the  ADS-B  links  could  not  see  or  be  seen  by  flight 
crews  using  a  CDTI  platform  with  the  other  ADS-B 
link.  Of  course,  unequipped  aircraft  also  would  not  be 
seen  on  CDTI  regardless  of  the  ADS-B  link  used.  A 


large  majority  of  crewmembers  commented  on  the 
enhanced  situation  awareness  benefit  that  would  be 
derived  if  all  aircraft  were  displayed.  As  would  be 
expected  with  CDTI  use,  some  flight  crews  also  noted 
that  they  would  have  to  continue  their  visual  search 
and  “see-and-avoid,”  out-the-window  surveillance 
procedure  to  detect  and  avoid  unequipped  aircraft  in 
a  mixed-equipage  environment.  One  of  the  flight 
crews  using  the  Advanced  Prototype  platform  com¬ 
mented  very  favorably  on  its  advanced  TCAS/ADS-B 
system,  which  displayed  both  ADS-B  and  transpon¬ 
der-equipped  aircraft,  essentially  providing  an  ap¬ 
proximation  of  the  full  equipage  that  crews  thought 
would  be  necessary  to  achieve  full  CDTI  benefits. 
Future  operational  evaluations  include  plans  to  dem¬ 
onstrate  a  multilateration-based  Traffic  Information 
System  -  Broadcast  (TIS-B)  that  would  enable  the 
display  of  all  transponder-equipped  aircraft  during 
surface  and  near-surface  observations. 

Display  Clutterf  Display  Resolutiony  Control  of 

Display  Range 

The  second  most  frequently  mentioned  issue  in  the 
System  category  was  display  clutter.  In  addition,  there 
were  several  related  comments  on  display  resolution, 
which  refers  to  the  ability  of  the  display  to  illustrate 
targets  in  sufficient  detail  so  that  flight  crews  can 
resolve  them  from  other  information.  Display-range 
control,  which  is  one  means  of  removing  unnecessary 
information  from  the  display,  also  was  mentioned  by 
flight  crews.  Examples  of  the  effects  of  clutter  men¬ 
tioned  by  the  crews  included  the  momentary  merging 
of  the  TTF  target  with  a  grouping  of  surface  targets  as 
TTF  approached  the  runway.  Flight  crews  reported 
that  this  complicated  tasks  associated  with  the  FAROA 
Air  application  for  the  airplane  that  was  following, 
and  made  it  more  difficult  to  determine  runway 
occupancy.  The  runway  occupancy  task  also  high¬ 
lighted  display  resolution  problems,  especially  when 
conducted  without  benefit  of  an  electronic  surface 
map.  Flight  crews  had  to  infer  from  their  own  runway 
alignment  and  positions  of  other  displayed  aircraft 
whether  or  not  TTF  was  on  the  runway.  The  com¬ 
ments  regarding  control  of  display  range  also  were 
related  to  tasks  associated  with  the  FAROA  Air  appli¬ 
cation.  These  comments  referenced  both  workload 
issues  and  display  size  and  resolution.  The  available 
display  scales  and  the  size  of  the  display  required 
manual  adjustment  of  range  to  allow  monitoring  of 
the  runway  environment.  The  most  suitable  scale 
(generally  agreed  to  be  1  or  2  miles)  had  to  be  selected 
after  passing  the  final  approach  fix  and  during  the 
final  descent  for  landing,  which  is  normally  when 
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flight  crews  are  devoting  full  attention  to  the  landing 
task.  Several  crews  noted  additional  workload  associ¬ 
ated  with  reconfiguring  the  CDTL 

Overall,  flight  crew  comments  about  clutter  indi¬ 
cated  that  they  desired  a  more  effective  means  to 
eliminate  unnecessary  information  and  configure  the 
CDTI  to  provide  the  information  necessary  to  sup¬ 
port  OpEval-2  applications. 

Display  Integration 

The  last  issue  in  the  System  category  mentioned 
frequently  by  flight  crews  was  display  integration. 
Flight  crews  from  the  CDTI  Build  2/4,  Advanced 
Prototype  and  Basic  Prototype  platforms  all  noted 
that  traffic  integration  with  the  flightdeck  navigation 
display  would  provide  the  most  effective  tactical  pic¬ 
ture  for  situation  awareness.  Additionally,  flight  crews 
using  the  Advanced  Prototype  platform  desired  inte¬ 
gration  of  the  approach  spacing  speed  commands 
with  the  Primary  Flight  Display  airspeed  tape  to 
reduce  the  mental  and  physical  workload  of  translat¬ 
ing  a  stand-alone  speed  command  into  a  speed  control 
response.  Flight  crews  using  the  CDTI  Build  2/4 
platform  became  aware  of  the  integration  issue  while 
performing  tasks  associated  with  the  Final  Approach 
Spacing  application.  They  noted  competition  for 
attention  between  the  Head-Up  Display  (HUD), 
which  was  providing  primary  flight  control  informa¬ 
tion,  and  the  centrally  located  CDTL  This  competi¬ 
tion  occurred  despite  a  recommended  technique  that 
suggested  the  PNF  only  attend  to  the  CDTL  Perhaps 
with  practice  flight  crews  may  find  it  easier  to  disre¬ 
gard  the  display  and  rely  on  the  PNF  conveyance  of 
display  information.  Without  the  benefit  of  practice, 
flight  crews  using  the  CDTI  Build  2/4  platform 
suggested  that  providing  spacing  data  on  the  HUD 
would  be  more  effective  than  verbal  transmission  of 
the  information  from  the  PNF. 

Crew  Resource  Management 
The  second-most  frequent  set  of  issues  mentioned 
in  the  structured  interview  were  related  to  Crew 
Resource  Management  (CRM).  This  category  included 
comments  on  workload,  procedures,  allocation  of  PF 
and  PNF  duties,  intra-cockpit  communication,  and 
situation  awareness.  Nearly  80%  of  CRM-related 
comments  from  flight  crews  using  the  CDTI  Build  2/ 
4  platform  related  to  workload.  These  comments 
included  mention  by  flight  crews  of  workload  being 
higher  than  that  in  normal  line  operations,  distraction 
from  other  tasks,  and  crew  management  issues  related 
to  extracting  information  from  the  CDTL 


Workload  and  Task  Interference 

Flight  crew  comments  about  workload  were  usu¬ 
ally  in  the  context  of  the  Final  Approach  Spacing 
application.  Crews  reported  that  the  level  of  perceived 
workload  depended  somewhat  on  the  distance  that 
needed  to  be  made  up  during  the  final  segment.  The 
technique  of  using  the  Pilot  not  flying  (PNF)  to 
provide  information  from  the  display  to  the  PF  ap¬ 
peared  to  be  effective  in  performance  terms,  but  it 
apparently  increased  workload,  primarily  for  the  PNF. 
Effects  on  the  PNF  included  a  heavier  checklist  man¬ 
agement  burden  when  it  was  necessary  to  suspend  an 
ongoing  checklist  to  provide  spacing  information, 
and  interference  with  the  monitoring  of  airspeed  and 
altitude  during  the  final  approach  segment  inside  the 
final  approach  fix.  Several  flight  crews  using  the  Build 
2/4  CDTI  platform  indicated  that  they  missed  some 
of  the  ATC  communications  due  to  PNF  workload. 

Flight  crews  using  the  Advanced  Prototype  plat¬ 
form  also  expressed  concern  about  a  higher  than 
expected  workload  when  using  the  advanced  spacing 
algorithm  speed  commands  during  the  Final  Ap¬ 
proach  Spacing  application.  However,  these  crews 
indicated  that  this  task  had  little  impact  on  crew 
interaction.  These  comments  may  be  due  to  the  inte¬ 
grated  speed  command,  which  placed  a  command  bug 
directly  on  the  speed  tape,  eliminating  the  require¬ 
ment  for  verbal  transfer  of  the  speed  command  data  by 
the  PNF.  The  high  reported  workload  for  the  Final 
Approach  Spacing  application  was  related  to  the  fre¬ 
quency  and  magnitude  of  speed  adjustments.  Under 
certain  conditions,  flight  crews  judged  the  frequency  of 
adjustment  to  be  too  great,  and  the  magnitude  too  large. 

A  crewmember  using  the  Basic  Prototype  platform 
noted  that  the  scenarios  within  the  OpEval-2  flight 
periods  induced  a  higher-than-normal  workload  due 
to  the  closed  traffic  patterns  and  repetitive  checklist 
requirements.  This  compressed  the  normal  arrival 
workload  into  a  much  shorter  time  period,  even 
without  the  additional  CDTI-related  spacing  tasks. 
As  noted  by  more  than  one  flight  crew,  successful 
integration  of  CDTI  procedures  with  ongoing 
flightdeck  tasks  will  require  good  management  skills 
supported  by  well-designed  and  explicit  procedures. 

Interestingly,  the  structured  interview  data  revealed 
that  flight  crews  using  the  MX-20  platform  gave  many 
CRM-related  comments;  but,  these  indicated  few 
effects  on  workload  and  no  tradeoffs  between  attend¬ 
ing  to  the  CDTI  and  other  cockpit  tasks.  Several 
factors  could  explain  the  different  crew  reactions 
based  on  CDTI  platform.  Flight  crews  using  the  MX- 
20  platform  were  operating  as  single-pilot  and  safety 
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pilot  crews.  However,  the  MX-20  pilots  generally 
were  most  familiar  with  single  pilot  operations  and 
were  not  given  specific  training  in  CRM,  or  how  to 
use  their  safety  pilot  resource  efficiently.  Therefore, 
although  CRM  seemed  to  be  a  difficult  task,  it  did  not 
seem  to  interfere  with  other  cockpit  operations.  In 
addition,  pilots  using  the  MX-20  platform  also  were 
not  assigned  the  spacing  tasks  that  generated  a  signifi¬ 
cant  proportion  of  workload  comments  from  the 
flight  crews  flying  other  CDTI  platforms. 

Finally,  many  flight  crews  using  the  MX-20  plat¬ 
form  were  flying  personally  owned  aircraft  and  were 
very  familiar  with  the  operation  of  their  traffic  dis¬ 
plays.  In  contrast,  many  flight  crews  using  other 
CDTI  platforms  did  not  see  the  platform  in  operation 
until  the  beginning  of  OpEval-2.  They  had  minimal 
training  on  the  CDTI  platform  and  no  formal  train¬ 
ing  on  integrating  CDTI  procedures  with  OpEval-2 
applications.  The  platform  was  being  flown  in  “nor¬ 
mal”  flight  operations,  but  the  OpEval-2  flight  peri¬ 
ods  were  not  equal  to  normal  revenue  operations 
experienced  by  line  flight  crews.  Hence,  these  data 
provide  a  snapshot  of  the  workload  that  existed  dur¬ 
ing  OpEval-2  conditions  and  further  research  and 
analysis  is  required  to  adequately  characterize  the 
workload  associated  with  CDTI  platforms  when  they 
are  used  during  normal  revenue  operations. 

Situation  Awareness 

Enhanced  situation  awareness  was  reported  by  flight 
crews  using  all  CDTI  platforms  and  was  a  consistently 
positive  finding  in  the  interview  statements.  Among 
the  reported  benefits  were  perceived  improvements  in 
tasks  associated  with  the  Final  Approach  Spacing 
application,  improved  awareness  of  local  traffic  situ¬ 
ations,  and  the  ability  to  determine  the  best  merge 
into  an  existing  traffic  flow.  For  example,  as  one 
crewmember  indicated,  being  able  to  view  the  loca¬ 
tion  of  the  downwind  to  base  turn  makes  it  possible  to 
efficiently  schedule  approach  briefings  and  checklist 
completion  without  guessing  how  much  time  is  avail¬ 
able.  For  surface  operations,  flight  crews  reported  that 
having  traffic  information  was  beneficial,  but  such 
information  would  be  of  greater  value  with  the  addi¬ 
tion  of  an  electronic  surface  map.  Although  flight 
crews  were  aware  of  a  workload  cost  imposed  by  the 
addition  of  CDTI  platforms,  most  indicated  that  this 
cost  might  be  mitigated  or  eliminated  with  more 
experience,  better  procedures  design,  and  training. 


Training 

Flight  crews  using  the  Build  2/4  platform  provided 
most  of  the  comments  in  the  training  category.  Sev¬ 
eral  comments  indicated  that  the  training  for  the 
OpEval-2  applications  was  inadequate  and  there  were 
many  suggestions  for  improvement.  The  most  fre¬ 
quent  comment  was  a  desire  for  hands-on  practice 
with  the  platform,  preferably  in  the  workload  context 
of  a  full  mission  simulation  or  line-oriented,  flight¬ 
training  scenario.  Short  of  this,  a  desktop  simulation, 
dedicated  CDTI  training  aids,  or,  as  suggested  by 
more  than  one  pilot  using  the  MX-20  platform,  a  CD- 
ROM  demonstration  of  acceptable  procedures  should 
be  provided.  Additional  training  would  be  very  ben¬ 
eficial  to  crew  performance  and  it  could  significantly 
reduce  workload.  Evidence  for  this  statement  comes 
from  the  frequent  observation  by  flight  crews  and 
observers  that  there  was  a  noticeable  learning  curve 
from  one  flight  period  to  the  next.  These  improve¬ 
ments  appeared  to  be  due  to  flight  crews’  exposure  to 
CDTI  platforms  and  OpEval-2  applications.  The 
exact  form  of  an  appropriate  training  program  must 
be  determined  by  operators  and  approved  by  the  FAA. 

ATC  Interaction 

Regardless  of  the  CDTI  platform,  the  majority  of 
flight  crew  responses  in  this  category  pointed  to  a 
need  for  exact  specification  of  procedures  for  flight 
crews  and  controllers  governing  the  use  of  CDTI. 
Advanced  Prototype  flight  crews  also  noted  that  it 
would  be  very  important  for  controllers  to  be  familiar 
with  CDTI  capabilities  and  limitations.  Spacing  tech¬ 
niques  by  flight  crews  using  CDTI  may  result  in 
observed  aircraft  behavior  that  differs  substantially 
from  controller  spacing  techniques.  Flight  crews  also 
noted  that  some  of  the  OpEval-2  restrictions  on 
CDTI-based  maneuvering  limited  their  ability  to 
effectively  use  CDTI  to  accomplish  spacing  objec¬ 
tives.  In  particular,  the  prohibition  on  lateral  maneu¬ 
vers,  combined  with  short  leg  lengths,  made  it  difficult 
to  achieve  spacing  goals  using  speed  control  alone. 
Flight  crews  using  the  Advanced  Prototype  platform 
indicated  that  it  would  be  very  important  to  have 
procedures  in  place  that  specify  when  and  where  flight 
crews  would  assume  responsibility  for  tasks  associated 
with  the  Final  Approach  Spacing  application.  Their 
comments  included  a  broad  range  of  suggestions, 
including  improved  phraseology,  explicit  spacing  as¬ 
signments,  and  the  possible  creation  of  ‘electronic 
flight  rules’  to  govern  the  integration  of  final  ap¬ 
proach  spacing  tools  with  existing  ATC  procedures. 
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Several  flight  crews  suggested  that  the  general  knowl¬ 
edge  of  the  traffic  situation  due  to  CDTI  reduced  the 
level  of  ATC  communication.  Others  reported  more 
accurate  visual  search  for  called  traffic  as  a  benefit. 

Applications 

Comments  on  applications  came  from  the  flight 
crews  using  the  CDTI  Build  2/4  and  Basic  Prototype 
platforms.  In  general,  they  indicated  that  Departure 
Spacing  was  easily  accomplished  and  that  Initial  Spac¬ 
ing  using  CDTI  was  accurate  and  provided  for  earlier 
detection  of  spacing  error  than  would  be  possible  with 
spacing  provided  by  ATC.  Flight  crew  comments  also 
indicated  that  the  absence  of  an  electronic  surface 
map  made  it  difficult,  especially  with  existing  display 
resolutions,  to  determine  directly  from  the  display 
whether  or  not  a  runway  was  occupied.  On  the  other 
hand,  observer  comments  indicated  that  flight  crews 
did  become  aware  of  possible  incursion  traffic  by 
referencing  CDTI,  which  resulted  in  increased  visual 
search  out  the  window  to  locate  traffic. 

Comments  by  flight  cre%vs  using  the  CDTI  Build 
2/4  platform  for  tasks  associated  with  the  Final  Ap¬ 
proach  Spacing  application  suggest  that  this  task  was 
difficult  under  OpEval-2  conditions.  The  restriction 
on  lateral  maneuvering  reduced  the  ability  of  flight 
crews  to  use  CDTI  for  establishing  position  to  com¬ 
plete  the  spacing  task.  When  forced  to  use  speed 
alone,  they  expressed  concern  about  the  possible  nega¬ 
tive  impact  on  a  stabilized  final  approach.  Finally, 
several  comments  related  to  the  need  for  an  electronic 
surface  map  to  support  the  Airport  Surface  Situation 
Awareness  application.  Flight  crews  commented  that 
although  the  display  of  other  traffic  is  useful,  without 
an  electronic  surface  map  there  is  little  support  for 
runway  incursion  prevention. 

Post-Flight  Flight  Crew  Questionnaires 

Two  post-flight  questionnaires,  one  for  the  MX-2G 
platform  and  one  for  the  three  other  CDTI  platforms, 
were  administered  after  flight  crews  completed  partici¬ 
pation  in  their  final  OpEval-2  flight  period.  The  ques¬ 
tionnaires  Included  statements  about  OpEval-2 
applications  and  CDTI  platform  usability.  These  state¬ 
ments  were  representative  of  the  following  topics: 

*  Training  -  previous  experience  with  and  operation  of 
platform. 

*  OpEval-2  Applications  -  use  of  CDTI  platform  for 
ASSA,  Departure  Spacing,  Traffic  Awareness,  Initial 
Approach  Spacing,  Final  Approach  Spacing,  and 
FAROA. 

*  Platform  Functions  -  ease  of  use  of  functions  during 
operation  (e.g..  Map  Range,  Target  Select,  etc). 


*  Platform  Color  and  Symbology  -  assessed  meaning- 
fulness  of  display  colors  and  symbols  during  opera¬ 
tion  of  the  platform. 

*  Platform  Features  -  assessed  utility  of  platform  func¬ 
tions,  as  well  as  flight  crew  perceptions  of  workload, 
heads-down  time  and  effect  on  decision  making. 

*  Location  and  Readability  -  assessed  location  and 
integration  of  the  platform  into  the  cockpit  display 
suite,  and  assessed  issues  associated  with  legibility. 

Analysis  of  data  from  the  questionnaires  included 
frequency  counts,  descriptive  statistics  and  analysis  of 
variance  on  summary  data,  which  was  derived  by 
combining  groups  of  ratings  for  OpEval-2  applica¬ 
tions.  In  addition,  factor  analyses  were  used  to  deter¬ 
mine  if  usability  ratings  for  usability  statements  could 
be  grouped  according  to  some  underlying  factor. 
Summary  data  was  computed  for  each  factor  and 
analysis  of  variance  was  performed  to  determine  if 
OpEval-2  platform  types  varied  along  these  factors. 

Responses  to  open-ended  questions  were  analyzed 
to  Identify  other  issues  that  flight  crewmembers 
thought  were  important.  Open-ended  responses  from 
the  two  questionnaires  were  aggregated  with  responses 
from  the  structured  interview  to  summarize  the  opera¬ 
tional  and  human  factors  issues  that  flight  crews  identi¬ 
fied  for  each  platform  type  and  OpEval-2  application. 

Flight  Crew  Training 

Thirty-four  percent  of  the  flight  crews  participat¬ 
ing  in  Op-Eval-2  had  some  previous  experience  flying 
a  CDTI-type  platform.  Approximately  82%  of  these 
flight  crews  stated  that  they  flew  CDTI  on  a  test 
flight.  This  experience  may  have  been  gained  during 
OpEval-1,  in  which  some  of  the  OpEval-2  flight 
crews  participated.  The  mean  number  of  hours  flown 
using  CDTI-type  platform  ranged  from  1 .0  hr  to  275*0 
hrs,  with  a  mean  of  19.1  hr  and  a  median  of  5.0  hr. 

Table  5  shows  the  different  methods  of  CDTI-type 
training  to  which  flight  crews  were  exposed.  The  I- 
LAB  training  consisted  of  simulation  training  at  MI¬ 
TRE  CAASD  in  preparation  for  OpEval-2.  OpEval-2 
training  included  the  briefings  provided  flight  crews 
immediately  before  OpEval-2  began.  Formal  training 
included  classroom  and  flight  instruction  (simula¬ 
tion).  Informal  training  included  reading  the  plat- 
form-operating  manual  and  other  training  included 
all  types  of  training  not  covered  by  the  training  types 
previously  listed.  Table  5  reveals  that  flight  crews  who 
participated  In  OpEval-2  received  most  of  their  train¬ 
ing  from  OpEval-2  briefings  and  from  informal  meth¬ 
ods,  such  as  reading  the  platform  operating  manual. 


21 


Table  5.  Summary  of  CDTI  training  listed  by  OpEval-2  flight  crews. 


Rating  of  Training  by  Flight  Crews 
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Figure  14.  Average  pilot  rating  for  each  OpEval-2  application  by  CDTI  platform  type. 


OpEval-2  Applications 

Flight  crews  rated  their  level  of  agreement  to  each 
questionnaire  statement  by  using  Likert-type  scales 
(l=Strongly  Disagree,  2=Somewhat  Disagree,  3=Nei- 
ther  Agree  Nor  Disagree,  4=Somewhat  Agree, 
5=Strongly  Agree).  Pilot  ratings  were  converted  to  a 
favorable  -  unfavorable  scale  to  simplify  interpreta¬ 
tion.  Ratings  less  than  2.75  were  considered  unfavor¬ 
able  and  ratings  greater  than  3.25  were  considered 
favorable. 

Figure  14  presents  the  average  pilot  rating  for  each 
OpEval-2  application.  The  ratings  are  classified  fur¬ 
ther  by  the  CDTI  platform  type.  The  average  rating 


for  an  application  was  derived  by  summing  all  ratings 
for  that  application  and  dividing  this  sum  by  the  total 
number  of  ratings  for  that  application.  This  method 
was  used  after  analysis  of  ratings  for  individual  ques¬ 
tionnaire  statements  showed  minimal  variation  in 
flight  crew  responses.  A  one-way  analysis  of  variance 
(ANOVA)  was  performed  for  each  application,  using 
CDTI  platform  type  as  the  independent  variable  and 
average  rating  for  each  OpEval-2  application  as  the 
dependent  variable. 
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Table  6  includes  the  results  of  post-hoc  tests  for 
each  of  the  OpEval-2  application  average  ratings.  The 
n  indicates  that  one  CDTI  platform  type  was  rated 
significantly  higher  than  the  CDTI  platform  type(s) 
with  U  in  the  same  row.  For  example,  the  n  and  U  in 
the  row  for  the  ASSA  application  indicate  that  the 
CDTI  Build  2/4  platform  was  rated  significantly 
higher  than  the  Basic  Prototype  platform.  The  MX- 
20  platform  was  used  only  for  the  ASSA  and  FAROA 
applications  and  to  evaluate  traffic  awareness.  It  was 
not  used  during  any  of  the  spacing  applications. 

ASSA 

The  ASSA  application  portion  of  the  questionnaire 
included  statements  that  elicited  flight  crew  responses 
about  how  well  the  CDTI  Build  2/4,  Basic  Prototype 
and  Advanced  Prototype  platform  functions  and  fea¬ 
tures  supported  surface  awareness,  taxiing,  and  acqui¬ 
sition  of  ground  traffic.  The  results  of  the  ANOVA 
indicated  that  the  highest  ratings  were  associated  with 
the  CDTI  Build  2/4  and  MX-20  platforms,  as  shown 


in  Figure  14.  The  flight  crews  agreed  somewhat  that 
these  two  platform  types  enhanced  ASSA.  Table  6 
reveals  that  both  the  CDTI  Build  2/4  and  MX-20 
platform  were  rated  significantly  higher  than  the 
Basic  Prototype  platform.  All  platform  types  received 
favorable  ratings  except  the  Basic  Prototype  platform. 
Examination  of  the  individual  item  responses,  how¬ 
ever,  indicated  that  most  flight  crews  found  clutter  to 
be  an  issue  for  the  ASSA  application. 

Departure  Spacing  (Not  Applicable  to  MX-20) 

The  Departure  Spacing  application  portion  of  the 
questionnaire  Included  statements  that  elicited  flight 
crews1  responses  about  how  well  the  CDTI  Build  2/4, 
Basic  Prototype  and  Advanced  Prototype  platform 
functions  and  features  supported  climb-out,  traffic 
awareness,  and  range  Identification  tasks  during  de¬ 
parture  spacing.  Figure  14  shows  that  the  CDTI  Build 
2/4  and  Advanced  Prototype  platform  received  favor¬ 
able  ratings.  Although  the  Basic  Prototype  was  not 
rated  unfavorable,  it  did  receive  the  lowest  overall 


Table  6.  Tukey  HSD  post-hoc  tests  for  flight  crew  questionnaire  summary  ratings  by  platform  type  and 
OpEval-2  application  or  Usability  Factor.1 
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1  All  post-hoc  comparison  tests  were  conducted  using  a  significance  criterion  of  0.05.  indicates  that  a  given  platfomi  type  was  rated 

significantly  higher  than  the  platform  type(s)  with  "4'”  in  the  same  row.  Within  rows,  platform  types  with  arrows  in  the  same  direction 
(e.g.,  CDTI  Build  2/4  and  MX-20  platforms  for  ASSA)  are  not  significantly  different,  “n.s”  indicates  that  a  given  platform  type  does  not 
differ  from  the  other  platform  types  in  the  same  row.  ”  is  used  to  indicate  that  the  MX-20  was  not  used  for  an  OpEval-2  application 
or  that  the  MX-20  was  not  rated  on  a  given  factor.  Cells  with  indicate  that  more  than  one  row  was  needed  to  explain  the 

relationship  between  the  ratings  for  platform  types.  For  example,  the  comparisons  for  the  “Color  and  Symbology”  factor  could  not  be 
represented  using  a  single  row.  As  shown  in  the  first  row  for  this  factor,  CDTI  Build  2/4  platform  ratings  were  significantly  higher  than 
those  of  the  Basic  and  Advanced  Prototypes,  However,  CDTI  Build  2/4  platform  ratings  did  not  differ  significantly  from  those  of  the 
MX-20,  which,  as  shown  in  the  second  row  for  this  factor,  were  significantly  higher  than  ratings  for  the  Basic  Prototype. 

2  The  CDTI  Build  2/4  platform  consistently  was  rated  higher  than  the  Advanced  Prototype  platform;  however,  statistical  tests  that  included 
the  latter  platform  had  very  low  power. 
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rating  for  the  Departure  Spacing  application.  Also, 
Table  6  reveals  that  the  ratings  for  the  Basic  Prototype 
platform  were  significantly  lower  than  those  for  CDTI 
Build  2/4  platform. 

Visual  Acquisition  and  Traffic  Awareness 

The  Visual  Acquisition  and  Traffic  Awareness  ap¬ 
plication  portion  of  the  questionnaire  included  state¬ 
ments  that  elicited  flight  crews’  responses  about  how 
well  the  CDTI  Build  2/4,  Basic  Prototype,  Advanced 
Prototype,  and  MX-20  platform  functions  and  fea¬ 
tures  supported  locating  and  identifying  proximate 
traffic.  Figure  14  shows  that  the  CDTI  Build  2/4  and 
MX-20  platform  received  very  favorable  ratings, 
whereas  the  Basic  Prototype  and  Advanced  Prototype 
platform  did  not.  Table  6  indicates  that  the  Traffic 
Awareness  ratings  associated  with  the  Basic  and  Ad¬ 
vanced  Prototype  platform  types  were  significantly  lower 
than  the  ratings  for  the  CDTI  Build  2/4  platform. 

Initial  Approach  Spacing  (Not  Applicable  to  MX-20) 

The  Initial  Approach  Spacing  application  portion 
of  the  questionnaire  included  statements  that  elicited 
flight  crews’  responses  about  how  well  the  CDTI 
Build  2/4,  Basic  Prototype,  and  Advanced  Prototype 
platform  functions  and  features  supported  target  se¬ 
lection,  station  keeping,  and  estimating  distance  and 
speed.  Although  Figure  14  shows  that  the  CDTI 
Build  2/4  and  the  Basic  Prototype  platform  received 
favorable  ratings,  the  CDTI  Build  2/4  platform  was 
rated  the  highest.  Table  6  indicates  that  the  rating  for 
the  CDTI  Build  2/4  platform  was  significantly  higher 
than  the  rating  associated  with  the  Basic  Prototype 
platform.  The  ratings  for  the  Basic  Prototype  and 
Advanced  Prototype  platform  were  equivalent. 

Final  Approach  Spacing  (Not  Applicable  to  MX-20) 

The  primary  objective  of  the  Final  Approach  Spac¬ 
ing  application  was  to  minimize  the  variability  in 
spacing  between  aircraft  on  approach.  The  Final  Ap¬ 
proach  Spacing  application  portion  of  the  question¬ 
naire  included  statements  that  elicited  flight  crews’ 
responses  about  how  well  the  CDTI  Build  2/4,  Basic 
Prototype,  and  Advanced  Prototype  platform  func¬ 
tions  and  features  supported  target  selection,  station 
keeping,  and  estimating  distance  and  speed.  Figure  14 
shows  that  the  ratings  for  this  application  look  very 
similar  to  the  ratings  for  the  Initial  Approach  Spacing 
application.  The  CDTI  Build  2/4  and  Basic  Proto¬ 
type  platform  received  favorable  ratings,  whereas  the 
Advanced  Prototype  platform  did  not.  Table  6 
indicates  that,  although  the  ratings  for  the  CDTI 


Build  2/4  platform  were  higher  than  those  for  the 
Basic  and  Advanced  Prototype  platform,  these  differ¬ 
ences  were  not  significant. 

FAROA 

The  FAROA  application  portion  of  the  question¬ 
naire  included  statements  that  elicited  flight  crews’ 
responses  about  how  well  the  CDTI  Build  2/4,  Basic 
Prototype,  Advanced  Prototype  and  MX-20  platform 
functions  and  features  supported  awareness,  target 
detection  in  the  runway  environment,  transition  pro¬ 
cedures,  and  estimation  of  runway  threshold.  The 
results  of  the  ANOVA  indicated  that  the  highest 
ratings  were  associated  with  the  CDTI  Build  2/4  and 
MX-20  platforms.  This  finding  is  shown  in  Figure  1 4. 
The  flight  crews  somewhat  agreed  that  these  two 
platform  types  enhanced  final  approach  runway  occu¬ 
pancy  awareness.  Table  6  reveals  that  ratings  for  the 
CDTI  Build  2/4  and  MX-20  platforms  were  signifi¬ 
cantly  higher  than  ratings  for  the  Basic  Prototype 
platform.  Ratings  of  the  Basic  Prototype  platform 
were  unfavorable.  In  fact,  80%  of  the  flight  crews  gave 
an  unfavorable  rating  of  this  platform  type. 

CDTI  Platform  Usability 

Exploratory  principal  components  analysis  (using 
varimax  rotation  with  Kaiser  normalization)  was  used 
to  analyze  the  pilot  ratings  of  statements  in  the  ques¬ 
tionnaire.  Principal  components  analysis  is  a  statisti¬ 
cal  technique  that  is  applied  to  a  set  of  variables,  such 
as  questionnaire  statements,  to  discern  which  vari¬ 
ables  in  the  set  form  coherent  subsets,  each  of  which 
are  relatively  independent  of  one  another.  Variables 
within  subsets  are  correlated;  however,  subsets  are 
largely  independent  of  other  subsets.  For  example,  a 
group  of  statements  within  a  section  of  the  flight  crew 
questionnaire  (e.g.,  Location  and  Readability  section) 
may  be  highly  correlated  and  independent  of  other 
groups  of  statements  within  the  same  section.  The 
correlation  among  statements  within  a  given  subset  is 
interpreted  as  an  underlying  component  or  factor  that 
can  be  used  to  describe  the  subset  of  statements. 
Principal  components  analysis  was  used  in  the  present 
context  to  reduce  a  large  number  of  questionnaire 
statements  to  a  smaller  number  of  usability  factors  on 
which  OpEvaI-2  platform  types  could  be  compared.  In 
addition,  the  definition  of  factors  yields  information 
that  can  be  used  to  provide  human  factors  guidance 
during  the  certification  process  (e.g.,  factors  become 
checklist  items  in  an  HF  supplement  to  a  TSO). 
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Four  principal  components  analyses  were  per¬ 
formed,  one  for  each  of  the  ergonomics  sections  in  the 
questionnaire.  The  results  of  these  analyses  are  sum¬ 
marized  in  the  following  list,  which  identifies  10 
factors; 

1 :  Display  functions 

2:  Consistent  and  acceptable  color  and  symbology 
3:  Enhanced  decision  making  and  task  completion 
4;  Understanding  of  display  information 
5:  Head-down  time  and  workload 
6:  Display  of  aircraft  speed  information 
7:  Readability  and  display  location 
8:  Button  labeling,  layout  and  size 
9:  Symbol  size 
10:  Button  cycling 

Summary  data  were  computed  for  each  factor,  and 
analysis  of  variance  was  performed  to  determine  if 
OpEval-2  platform  types  varied  along  these  factors. 
Figure  15  presents  the  average  pilot  rating  for  each 
usability  factor.  The  ratings  are  further  classified  by 
platform  type.  The  average  rating  for  a  given  factor 
was  derived  by  summing  all  ratings  for  that  factor  and 
dividing  by  the  total  number  of  ratings  for  that  factor. 

Factor  1:  Display  Functions  (Not  Applicable  to  MX - 
20).  Flight  crews  were  asked  to  rate  the  ease  of  use  of 
particular  display  functions  (e.g,,  display  mode,  tar¬ 
get  select,  etc.)  using  a  five-point,  Likert-type  scale 
(1=  Difficult  to  use,  5=Easy  to  use)  for  questionnaire 
items  113  -  120.  Figure  15  shows  that  flight  crews 
rated  the  CDTI  Build  2/4  platform  easiest  to  use 
relative  to  the  Basic  and  Advanced  Prototype  plat¬ 
form.  Table  6  reveals  that  the  CDTI  Build  2/4  plat¬ 
form  was  rated  significantly  easier  to  use  than  the 
Basic  Prototype  platform. 

Factor  2:  Color  and  Symbology .  The  Color  and 
Symbology  portion  of  the  questionnaire,  which  in¬ 
cluded  items  122  -  128,  elicited  flight  crews'  re¬ 
sponses  about  the  appropriateness  of  the  color  and 
symbology  used  in  the  displays  of  the  CDTI  Build  2/ 
4,  Basic  Prototype,  Advanced  Prototype  and  MX-20 
platform.  These  items  ranged  from  symbology  scaling 
issues  to  consistency  of  color  relative  to  other  flightdeck 
displays.  Figure  15  shows  that  the  ratings  for  this 
usability  factor  were  favorable  for  the  CDTI  Build  27 
4  and  MX-20  platforms  and  unfavorable  for  the  Basic 
and  Advanced  Prototype  platforms. 

Table  6  indicates  that  ratings  for  the  CDTI  Build 
2/4  and  MX-20  platforms  were  significantly  higher 
than  the  ratings  for  the  Basic  Prototype  platform.  In 


addition,  the  ratings  for  the  CDTI  Build  2/4  platform 
were  significantly  higher  than  the  ratings  for  the 
Advanced  Prototype  platform.  Written  observations 
suggested  that,  overall,  some  flight  crews  reported  diffi¬ 
culty  in  color  discrimination  between  cyan  and  green. 

Factors  3-6:  Display  Functions .  Factor  analyses  of 
pilot  ratings  defined  four  underlying  factors  in  this 
section  of  the  questionnaire.  Factor  3  included  ques¬ 
tionnaire  items  142,  143  and  146,  and  this  factor 
identified  decision  making  and  task  completion  is¬ 
sues.  Figure  1 5  shows  that  ratings  for  the  CDTI  Build 
2/4,  Advanced  Prototype  and  MX-20  platforms  all 
were  favorable.  The  rating  for  the  Basic  Prototype 
platform  was  unfavorable.  The  ANOVA  results  in 
Table  6  revealed  that,  for  factor  3,  the  CDTI  Build  21 
4  and  MX-20  platforms  were  rated  significantly  more 
favorable  than  the  Basic  Prototype  platform. 

Factor  4  included  questionnaire  items  132  -135, 
and  was  defined  as  types  of  display  information  (e.g., 
data  tags,  traffic  altitudes,  etc).  This  factor  was  not 
applicable  to  the  MX-20  platform.  Table  6  shows  that 
the  ratings  for  the  CDTI  Build  2/4,  Advanced  Proto¬ 
type,  and  Basic  Prototype  platforms  were  favorable. 
Table  6  indicates  that,  for  factor  4,  the  CDTI  Build  2/ 
4  platform  was  rated  significantly  more  favorable  than 
the  Basic  Prototype  platform. 

Factor  5  included  questionnaire  items  136,  144 
and  145,  and  this  factor  revealed  attentional  issues 
(e.g.,  head-downtime  and  workload).  Figure  15  shows 
that  only  the  MX-20  platform  was  rated  favorably  for 
this  factor.  Ratings  for  the  CDTI  Build  2/4,  Basic 
Prototype,  and  Advanced  Prototype  platforms  were 
unfavorable.  Table  6  shows  that,  for  Factor  5,  the 
MX-20  platform  was  rated  significantly  more  favor¬ 
able  than  the  Basic  Prototype  platform.  These  find¬ 
ings  may  indicate  that,  relative  to  the  MX-20  platform, 
the  CDTI  Build  2/4,  Basic  Prototype  and  Advanced 
Prototype  platforms  were  more  novel  to  flight  crews. 
Regardless  of  the  explanation,  issues  related  to  atten¬ 
tion  should  be  explored  and  attempts  should  be  made 
to  mitigate  the  possibility  of  flight  crews  attending 
too  much  to  OpEval-2  platform  types. 

Factor  6  included  questionnaire  items  131,  137 
and  139.  These  items  focused  on  display  of  aircraft 
speed  and  vector  information  (e.g.,  ground  speed 
tag).  This  analysis  was  not  applicable  to  the  MX-20 
platform.  Figure  15  shows  that  ratings  for  the  CDTI 
Build  2/4  platform  were  favorable,  whereas  ratings  for 
the  Basic  Prototype  and  Advanced  Prototype  platforms 
were  unfavorable.  Table  6  indicates  that,  for  factor  6, 
the  CDTI  Build  2/4  platform  was  rated  significantly 
more  favorable  than  the  Basic  Prototype  platform. 
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1.  Easy-to-use  display 
functions 

2.  Consistent  and 
acceptable  colorand 
symbology 

3.  Enhanced  decision 
making  and  task  completion 

4.  Understood  display 
info  rmatio  n 


5.  No  increase  in  head- 
down  time  and  workload 


6.  Display  of  aircraft  speed 
info rmation  was  useful 


7.  Acceptable  readability 
and  display  location 


8.  Acceptable  button 
labeling,  layout  and  size 


9.  Acceptable  symbol  size 


10.  Acceptable  button 
cycling  (e.g.,  target  and 
range  selection) 


Strongly  Somewhat  Neither  Agree  Somewhat  Strongly 

Disagree  Disagree  Nor  Disagree  Agree  Agree 


Figure  15.  Average  pilot  rating  for  each  usability  factor  by  platform  type. 


Factors  7-10:  Location  and  Readability.  Factor  analy¬ 
ses  of  pilot  ratings  also  defined  four  underlying  fac¬ 
tors  in  this  section  of  the  questionnaire.  Factor  7 
consisted  of  questionnaire  items  148,  150,  157-  159, 
and  1(53  -  166.  This  factor  seemed  to  reveal  issues 
related  to  readability  and  display  location.  Figure  15 
shows  that  ratings  for  the  CDTI  Build  2/4  and  MX- 
20  platforms  were  favorable,  whereas  the  ratings  for 
the  Basic  Prototype  and  Advanced  Prototype  platforms 
were  unfavorable.  The  ANOVA  results  in  Table  6  re¬ 
vealed  that,  for  factor  7,  ratings  for  the  MX-20  platform 
were  significantly  more  favorable  than  the  other  three 
platforms.  In  addition,  the  ratings  for  the  CDTI  Build 
2/4  platform  were  significantly  more  favorable  than  the 
Basic  and  Advanced  Prototype  platforms. 

Factor  8  was  defined  by  questionnaire  items  153  - 
156  and  167,  all  of  which  identified  control  button 
features  (e.g.,  button  labeling,  button  layout,  and 
button  size).  Figure  1 5  again  shows  that  the  ratings  for 
the  CDTI  Build  2/4  and  MX-20  platforms  were 
favorable,  whereas  the  ratings  for  the  Basic  Prototype 
and  Advanced  Prototype  platforms  were  unfavorable. 
The  ANOVA  results  in  Table  6  revealed  that,  for 


factor  8,  ratings  for  the  MX-20  platform  were  signifi¬ 
cantly  more  favorable  than  the  ratings  for  the  Basic 
and  Advanced  Prototype  platforms. 

Factor  9  consisted  of  questionnaire  items  149  and 
160  -162.  These  items  identified  issues  related  to 
display  symbol  size.  The  analysis  was  not  applicable  to 
the  MX-20  platform.  Figure  1 5  shows  that  the  CDTI 
Build  2/4  platform  was  rated  as  favorable.  Table  6 
shows  that,  for  factor  9,  the  CDTI  Build  2/4  platform 
was  rated  significantly  more  favorable  than  the  Basic 
Prototype  platform. 

Factor  10  included  questionnaire  items  151  and 
152.  This  factor,  although  difficult  to  interpret,  ap¬ 
peared  to  characterize  issues  associated  with  button 
cycling  during  target  and  range  selection.  The  analysis 
was  not  applicable  to  the  MX-20  platform.  Figure  15 
shows  that  ratings  for  the  CDTI  Build  2/4  platform 
were  favorable,  whereas  ratings  for  the  Advanced 
Prototype  platform  were  unfavorable.  Table  6  indi¬ 
cates  that,  for  factor  1 0,  the  CDTI  Build  2/4  platform 
was  rated  significantly  more  favorable  than  both  the 
Basic  and  Advanced  Prototype  platforms. 
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CONCLUSIONS 

The  OpEval-2  flightdeck  observer  data  supple¬ 
ments  a  corpus  of  evidence  that  contributes  to  the 
identification  and  resolution  of  human  factors  issues 
associated  with  the  use  of  CDTI  by  flight  crews.  The 
following  conclusions  and  recommendations,  which 
are  organized  by  OpEval-2  applications  and  CDTI 
certification  issues,  are  drawn  from  analysis  of  the 
flightdeck  observer  data. 

OpEval-2  Applications 

ASSA 

While  navigating  the  airport  surface  along  complex 
routes,  flight  crews  who  used  electronic  surface  maps 
were  able  to  easily  perform  taxi  operations  and  iden¬ 
tify  taxiway  locations  relative  to  the  crews  who  used 
paper  surface  maps.  Nearly  67%  of  the  flight  crews 
with  electronic  surface  maps  used  them  for  awareness 
of  ownship  position  on  the  airport  surface,  and  they 
were  able  to  identify  their  location  on  the  airport 
surface  whereas  crews  with  a  paper  surface  map  were 
not.  All  but  one  of  the  flight  crews  with  an  electronic 
surface  map  used  the  CDTI  to  support  surface  taxi 
operations  at  least  some  of  the  time.  Also,  flight  crews 
unanimously  agreed  that  awareness  of  airport  surface 
operations  was  enhanced  when  the  CDTI  platform 
included  an  electronic  surface  moving  map.  Taken 
together,  these  findings  Indicate  that  a  CDTI  show¬ 
ing  the  position  of  own  aircraft  relative  to  an  airport 
map  appears  to  improve  geographical  and  situation 
awareness. 

Flightdeck  observer  data  revealed  that  flight  crews 
with  electronic  surface  moving  maps  committed  fewer 
errors  than  did  flight  crews  with  paper  surface  maps. 
Furthermore,  when  a  taxi  error  was  made,  flight  crews 
with  electronic  surface  maps  also  were  able  to  rejoin 
their  assigned  structured  routes  more  readily  than 
those  with  paper  surface  maps.  Although  statistical 
analysis  of  errors  was  not  possible  because  there  were 
too  few,  this  evidence  suggests  that  flight  crew  aware¬ 
ness  can  be  enhanced  by  a  CDTI  with  an  electronic 
surface  map.  This  evidence  is  compelling  given  that 
flight  crews  with  paper  surface  maps  were  much  more 
familiar  with  the  airport  used  in  this  study  than  those 
with  electronic  surface  maps. 

Flightdeck  observers  reported  that  a  higher  per¬ 
centage  of  crews  with  an  electronic  surface  map  used 
it  to  enhance  awareness  of  the  airport  surface  relative 
to  those  with  a  paper  surface  map.  This  enhanced 
awareness  was  accompanied  by  an  Increase  in  the  use 
of  the  electronic  surface  map  to  assess  traffic  location 


on  the  airport  surface,  possible  conflicts  during  taxi 
along  the  structured  routes,  locating  TTF,  and  for 
general  traffic  awareness.  Post-flight  interviews  indi¬ 
cated  that  the  addition  of  traffic  information  was 
valuable  for  the  ASSA  application,  and  that  this  infor¬ 
mation  was  enhanced  with  the  addition  of  an  elec¬ 
tronic  surface  map.  When  asked  whether  CDTI  aided 
in  traffic  awareness,  flight  crews  who  used  an  elec¬ 
tronic  surface  map  agreed  more  strongly  that  it  did 
relative  to  flight  crews  with  a  paper  surface  map. 

In  their  responses  to  the  post-flight  questionnaires, 
all  flight  crews  agreed  that  CDTI  (with  and  without 
an  electronic  surface  moving  map)  enhanced  airport 
surface  situation  awareness.  Overall,  flight  crews  us¬ 
ing  the  MX-20  platform,  which  included  the  moving 
map,  did  provide  higher  ratings  for  ASSA,  but  the 
difference  was  not  statistically  significant. 

Flightdeck  observer  data  indicated  that  the  CDTI 
function  known  as  Target  Selection  was  used  by 
approximately  the  same  percentage  of  crews  with 
electronic  surface  maps  as  those  with  paper  maps. 
The  function  was  used  to  aid  surface  navigation.  The 
Ground  Track  Vector  function  was  used  by  few  crews 
during  any  task,  regardless  of  whether  these  crews  had 
an  electronic  surface  map.  The  Range  function  was 
used  twice  as  often  by  crews  with  electronic  surface 
maps  as  crews  with  paper  surface  maps.  These  data 
indicate  that,  for  the  ASSA  application,  the  Target 
Selection  function  may  be  valuable  for  CDTI  with  or 
without  an  electronic  surface  map,  the  Range  func¬ 
tion  may  be  particularly  valuable  for  CDTI  with  an 
electronic  surface  map,  and  the  Ground  Track  Vector 
function  does  not  appear  to  be  useful.  Flight  crew 
comments  from  post-flight  Interviews  indicated  that 
use  of  the  CDTI  for  the  ASSA  application  could  be 
enhanced  by  developing  an  effective  means  to  control 
clutter,  displaying  runway  and  taxiway  markings, 
using  a  track-up  map  orientation,  providing  NOTAM 
data  (e.g.  closed  runways  and  taxiways),  and  includ¬ 
ing  a  pan  and  zoom  function. 

Flight  crews  were  given  a  high-workload,  surface 
navigation  task  during  which  their  navigation,  geo¬ 
graphic  and  traffic  awareness  were  evaluated.  Flight 
crews  that  used  CDTI  platforms  with  electronic  air¬ 
port  surface  maps  and  those  who  used  paper  surface 
maps  both  agreed  that  It  was  easy  to  get  oriented  to  the 
CDTI.  These  crews  also  reported  that  CDTI  aided 
the  tasks  of  locating  and  determining  the  positions  of 
traffic,  and  that  It  aided  their  understanding  of  ATC 
communications.  These  findings  imply  that  CDTI 
reduced  the  workload  of  flight  crews  during  airport 
surface  operations.  Additionally,  there  was  almost 
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unanimous  agreement  by  flight  crews  on  the  need  for 
an  electronic  surface  map  on  the  CDTI  to  support  the 
ASSA  application. 

Departure  Spacing 

The  Departure  Spacing  application  demonstrated 
that  flight  crews  using  CDTI  and  controllers  using 
current  procedures  without  ADS-B  could  minimize 
spacing  variation  and  that  it  would  not  affect  their 
workload.  For  example,  post-flight  interview  data 
suggested  that  flight  crews  generally  thought  that 
tasks  in  the  Departure  Spacing  application  were  easy 
to  perform.  This  finding  suggests  that  the  Departure 
Spacing  application  would  not  affect  flight  crew 
workload.  However,  more  research  is  required  to 
determine  the  impact  on  airport  capacity  and  safety  of 
increased  “runway  ownership  time,”  which  represents 
the  time  allotted  by  the  controller  to  the  flight  crew  to 
maneuver  the  aircraft  onto  the  runway  and  complete 
the  departure  roll  after  receiving  a  take  off  clearance 
from  the  controller.  During  this  time,  the  runway  is 
unavailable  to  the  controller  for  vehicle  or  other 
aircraft  operations  (e.g.,  runway  crossing,  landings, 
etc.).  Additionally,  there  is  an  increase  in  the  potential 
for  an  accident  because  the  aircraft  remains  on  the 
active  runway  until  appropriate  spacing  is  achieved. 

Initial  and  Final  Approach  Spacing 

During  the  Initial  Approach  Spacing  application, 
flightdeck  observers  reported  high  rates  of  compli¬ 
ance  with  OpEval-2  recommended  CDTI  function 
settings  for  Range  Ring  (79%),  Vector  Time  (86%) 
and  Select  Target  (78%).  In  many  cases,  these  settings 
were  made  previously  and  simply  retained  by  flight 
crews.  Although  adherence  to  these  recommenda¬ 
tions  does  not  necessarily  validate  their  use  for  initial 
approach  spacing,  it  at  least  implies  that  the  flight 
crews  did  not  prefer  other  settings.  These  findings 
indicate  that  for  initial  approach  spacing,  flight  crews 
were  able  to  use  the  CDTI  with  the  range  ring,  vector 
time,  and  target  select  functions.  It  should  be  noted 
that  further  work  is  needed  before  initial  approach 
spacing  can  be  performed  by  flight  crews  in  the  NAS. 

Flightdeck  observers  also  found  that  85%  of  flight 
crews  used  the  Select  Target  function  during  the  Final 
Approach  application.  Flight  crews  and  observers 
both  reported  that  this  function  was  the  crews’  pri¬ 
mary  source  of  information.  Only  half  the  crews 
complied  with  the  Vector  Time  setting  recommenda¬ 
tion  and  this  finding  likely  was  due  to  the  incompat¬ 
ibility  of  the  vector  indications  (i.e.,  time)  with  the 


distance-based  information  provided  for  the  spacing 
task  (i.e.,  distance).  The  recommended  Range  Ring 
setting  was  observed  82%  of  the  time  during  the  Final 
Approach  Spacing  application. 

Analysis  of  flightdeck  observer  data  indicated  that 
pilot  workload  during  the  Initial  and  Final  Approach 
spacing  applications  varied  by  CDTI  platform  type. 
Cooper-Harper  workload  ratings  for  flight  crews  us¬ 
ing  the  CDTI  Build  2/4  CDTI  platform  revealed  that 
mental  effort  was  required  to  attain  acceptable  perfor¬ 
mance.  Ratings  associated  with  the  Advanced  Proto¬ 
type  platform  indicated  that  high  mental  effort  was 
needed  to  attain  acceptable  performance.  Ratings  for 
the  Basic  Prototype  platform  indicated  that  maxi¬ 
mum  mental  effort  was  required  to  avoid  large  or 
numerous  errors.  Although  these  results  indicate  that 
the  implementation  of  the  Basic  Prototype  platform 
resulted  in  excessive  workload  levels,  design  efforts  to 
reduce  the  workload  associated  with  all  platform 
types  would  be  beneficial  to  flight  crews. 

During  Final  Approach  Spacing,  flight  crews  using 
the  CDTI  Build  2/4  platform  reported  competition 
for  attention  between  the  CDTI  and  the  HUD,  which 
provided  primary  flight  control  information.  They 
commented  in  interviews  that  PNF  workload  was 
higher  than  would  be  acceptable  in  line  operations, 
and  that  use  of  the  CDTI  caused  distraction  from 
other  tasks,  such  as  hearing  ATC  communications, 
monitoring  airspeed  and  altitude,  and  checklist 
completion.  Written  comments  by  flight  crews  sug¬ 
gested  that  the  use  of  CDTI  resulted  in  two  of  the  five 
checklist  suspensions  and  five  of  the  ten  checklist 
interruptions. 

Flight  crews  using  the  Advanced  Prototype  plat¬ 
form  reported  higher  than  expected  workload  during 
the  Final  Approach  Spacing  application.  The  higher 
workload  was  attributed  to  the  frequency  and  magni¬ 
tude  of  speed  adjustments  required  by  the  advanced 
spacing  algorithm  commands.  Reducing  the  frequency 
and  magnitude  of  these  speed  commands  may  reduce 
the  mental  effort  currently  required  to  achieve  accept¬ 
able  performance.  The  relay  of  spacing  information 
from  PNF  to  PF  created  higher  workload  for  the  PNF. 
This  issue  could  be  mitigated  by  relocating  the  spacing 
information  within  the  PF’s  primary  field  of  view 
where  the  PF  would  be  able  to  access  it  directly.  Even 
with  this  solution,  the  CDTI-related  responsibilities 
for  the  PF  and  PNF  need  to  be  integrated  into  existing 
flightdeck  checklist  procedures  for  the  respective 
phases  of  flight. 
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FAROA 

The  FAROA  Air  application  required  flight  crews 
to  use  CDTI  while  on  final  approach  to  determine 
whether  the  runway  was  occupied.  During  some  re¬ 
stricted  low  approaches,  vehicles  were  positioned  on 
the  runway.  Eight  flight  crews  with  electronic  surface 
maps  identified  vehicles  were  occupying  the  runway, 
whereas  none  of  the  crews  with  paper  surface  maps 
were  able  to  identify  these  vehicles.  Although  these 
data  appear  to  indicate  that  a  CDTI  with  an  electronic 
surface  map  enhances  the  ability  of  flight  crews  to 
detect  runway  intrusions,  an  appropriate  comparison 
would  be  needed  to  establish  the  efficacy  of  using 
CDTI  with  a  paper  surface  map  for  FAROA  Air. 

In  their  responses  to  the  post-flight  questionnaire, 
the  flight  crews  using  the  CDTI  Build  2/4  and  MX- 
20  platforms  agreed  that  the  CDTI  supported  the 
FAROA  Air  application.  The  MX-20  platform  in¬ 
cluded  an  electronic  surface  map,  whereas  the  CDTI 
Build  2/4  platform  did  not.  Flight  crews  using  a 
CDTI  platform  that  included  an  electronic  surface 
map  agreed  that  the  CDTI  aided  in  detecting  traffic 
crossing  the  runway,  traffic  near  the  runway,  traffic 
on  the  runway,  and  gauging  when  traffic  exited  the 
runway.  Flight  crews  using  a  CDTI  with  a  paper 
surface  map  agreed  that  the  CDTI  aided  in  determin¬ 
ing  when  traffic  was  on  the  runway.  Data  from  post¬ 
flight  interviews  indicated  that  flight  crews  noted 
difficulty  in  determining  runway  occupancy  directly 
from  the  electronic  surface  map.  Anecdotal  data  from 
flightdeck  observers'  suggested  that  flight  crews  be¬ 
came  aware  of  possible  incursion  traffic  by  reference 
to  the  CDTI  without  an  electronic  surface  map,  after 
which  they  increased  their  visual  search  out  the  win¬ 
dow  to  locate  the  traffic. 

Overall,  flight  crews  generally  agreed  that  CDTI 
aided  in  taxiway,  runway,  final  approach,  and  depar¬ 
ture  traffic  awareness  during  takeoff.  Additionally, 
they  agreed  that  the  color  change  from  brown  (on  the 
surface)  to  green  (when  airborne)  aided  in  awareness 
of  aircraft  status.  Flightdeck  observers  reported  evi¬ 
dence  indicating  that  substantially  more  crews  with 
an  electronic  surface  map  than  crews  without  an 
electronic  surface  map  used  CDTI  to  assess  runway 
occupancy  as  well  as  the  approach  and  the  departure 
corridors.  These  findings  indicate  that  surface  map 
capability  appears  to  increase  the  use  of  CDTI  for 
Final  Approach  Runway  Occupancy  Awareness 
(FAROA)  Air  and  Ground. 

The  CDTI  proved  valuable  for  seeing  properly 
equipped  aircraft  on  or  near  the  runway  during  the 
takeoff  and  approach  phases  of  flight.  However,  as  the 


TTF  approached  the  runway,  the  symbol  that  repre¬ 
sented  it  on  the  display  merged  momentarily  with 
other  aircraft  symbols  that  were  grouped  as  surface 
targets.  Also,  the  absence  of  an  electronic  airport 
surface  map  and  the  inability  to  determine  the  precise 
position  of  traffic  relative  to  the  runway  environ¬ 
ments  were  problematic  for  flight  crews  on  final 
approach.  Many  flight  crews  overcame  these  prob¬ 
lems  by  using  the  Range  function  to  manually  adjust¬ 
ing  CDTI  range  to  one  or  two  miles  after  passing  the 
final  approach  fix,  which  is  normally  a  time  when 
crews  devote  full  attention  to  landing. 

Flightdeck  observers  also  noted  that  the  flight 
crews  performing  the  FAROA  Ground  and  Air  appli¬ 
cations  with  electronic  surface  maps  used  the  CDTI 
to  evaluate  approach  and  departure  corridors,  and 
runways  for  possible  conflicts.  During  final  approach 
when  flight  crews  were  performing  the  FAROA  Air 
application,  observers  reported  that  crews  with  an 
electronic  surface  map  used  different  surface  map 
modes.  Although  many  crews  used  the  Compass  Rose 
mode  and  some  used  the  Arc  mode,  most  crews  with 
a  paper  surface  map  used  the  Arc  mode.  The  Range 
function  was  used  to  adjust  the  map  range  by  most 
crews,  especially  those  with  electronic  surface  maps.  A 
relatively  small  number  of  crews  with  and  without 
electronic  surface  maps  used  the  Ground  Track  Vec¬ 
tor  function  for  runway  occupancy  awareness  during 
approach  to  display  target  speed.  These  data  indicate 
that,  for  the  FAROA  Air  application,  Track-up  360 
mode  and  the  Range  function  may  be  used  with  an 
electronic  surface  map  more  than  other  modes.  Arc 
mode  may  be  used  without  an  electronic  surface  map 
more  than  other  modes,  and  that  the  Ground  Track 
Vector  function  may  be  used  both  with  and  without 
the  electronic  surface  map. 

Visual  Acquisition  and  Traffic  Awareness 

As  much  as  possible,  flight  operations  using  CDTI 
in  a  mixed  equipage  environment  should  include  the 
display  of  all  aircraft  on  the  CDTI.  In  post-flight 
structured  interviews,  a  large  majority  of  flight  crews 
commented  that  the  display  of  all  aircraft  would  be 
necessary  to  achieve  the  full  benefit  of  CDTI.  One 
crewmember  using  the  Advanced  Prototype  platform, 
which  displayed  all  OpEval-2  aircraft,  commented 
favorably  on  the  utility  of  this  feature.  Some  flight 
crews  suggested  that  they  would  be  required  to  main¬ 
tain  a  robust  visual  search  for  aircraft  that  are  not 
displayed  In  mixed  equipage  conditions. 
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CDTI  Certification 

One  goal  of  the  OpEval-2  test  was  to  provide 
information  to  avionics  manufacturers  and  the  FAA 
Aircraft  Certification  Office  to  support  certification 
of  CDTI.  For  air  transport  category  aircraft,  the 
primary  certification  requirements  are  defined  by  the 
Code  of  Federal  Regulations  14  Part  25.  Part  25 
addresses  two  primary  issues:  (1)  Does  the  platform 
perform  the  intended  function?  and  (2)  Could  the 
platform  lead  to  hazardous  situations?  The  following 
paragraphs  summarize  flightdeck  observer  data  from 
OpEval-2  that  will  be  used  to  address  the  two  primary 
certification  issues  contained  in  Part  25. 

Usability 

Overall,  flight  crews  rated  the  CDTI  Build  2/4  plat¬ 
form  functionality  as  easy  to  use  and  agreed  that  it 
enhanced  decision  making  and  task  completion.  Display 
information  was  considered  understandable;  symbol 
size,  symbology,  button  cycling  were  considered  accept¬ 
able;  and,  use  of  color  was  considered  consistent  and 
acceptable.  Responses  to  the  flight  crew  questionnaire 
for  the  MX-20  platform  also  indicated  that  it  enhanced 
decision-making  and  task  completion,  the  readability 
and  display  location,  and  that  the  button  labeling, 
layout,  and  size  were  acceptable. 

The  Advanced  and  Basic  Prototype  CDTI  platform 
were  developed  specifically  for  OpEval-2.  The  Advanced 
Prototype  platform  readability  and  display  location  were 
rated  unfavorably;  particularly  its  button  labeling,  lay¬ 
out,  and  size.  The  Basic  Prototype  platform  readability 
and  display  location  also. were  rated  unfavorably.  These 
responses  may  be  related  to  the  display-control  interface 
and  not  the  actual  display  interface.  Flight  crew  ques¬ 
tionnaire  responses  suggested  that  workload  and  head- 
down  time  increased  most  with  the  Advanced  and  Basic 
Prototype  platforms,  although  responses  indicated  that 
they  increased  with  the  CDTI  Build  2/4  platform  as 
well.  Collectively,  these  findings  indicate  that  several 
usability  issues  remain  with  the  Basic  and  Advanced 
Prototype  platforms,  which  is  to  be  expected  given  that 
both  systems  were  still  in  the  developmental  stage. 

CDTI  in  Flight  Crew  Field  of  View 

An  important  certification  issue  pertains  to  whether 
the  CDTI  platform  should  be  included  in  the  flight 
crews’  primary  scan  pattern.  Aircraft  participating  in 
OpEval-2  had  the  CDTI  platform  mounted  in  the 
front  panel  of  the  flightdeck.  The  platform  appeared 
to  be  scanned  easily  for  all  applications  tested.  How¬ 
ever,  none  of  the  aircraft  had  the  information  from 


CDTI  integrated  with  the  Electronic  Horizontal  Situ¬ 
ation  Indicator  or  a  Head-up  Display  (HUD),  which 
are  in  the  flight  crews’  primary  scan  pattern.  Flight 
crews  commented  on  this  fact  and  they  noted  that 
traffic  integration  with  the  NAV  display  would  pro¬ 
vide  the  most  effective  tactical  information  for  situa¬ 
tion  awareness.  Furthermore,  flight  crews  commented 
that  the  CDTI  platform  was  out  of  the  primary  scan 
pattern  during  the  Final  Approach  application.  Ap¬ 
parently,  the  HUD,  which  was  providing  primary 
flight  control  information,  and  the  centrally  located 
CDTI  competed  for  flight  crews’  attention.  This 
phenomenon  occurred  despite  a  recommended  tech¬ 
nique  that  instructed  only  the  PNF  to  attend  to  the 
CDTI  platform.  Finally,  some  flight  crews  noted  that 
it  was  difficult  to  observe  CDTI  information  from 
other  locations  on  the  flightdeck. 

Display  Presentation 

Another  important  certification  question  pertains 
to  CDTI  platform  display  size  and  other  display 
attributes  that  supported  each  OpEval-2  application. 
Data  relevant  to  these  issues  were  collected  during 
OpEval-2  using  post-flight,  structured  interviews  and 
flight  crew  questionnaires.  Overall,  flight  crews  listed 
the  following  changes,  which  they  thought  would 
enhance  display  attributes  such  that  the  CDTI  plat¬ 
form  could  be  used  efficiently  for  the  applications 
tested  during  OpEval-2. 

•  A  track-up,  surface  moving  map  and  taxiway  labels 
that  are  always  in  view  and  properly  oriented. 

•  Better  color  differentiation-  Selected  target  vs.  back¬ 
ground;  also,  written  observations  suggested  that 
some  flight  crews  reported  difficulty  in  discriminat¬ 
ing  between  cyan  and  green. 

•  Display  feature  priority  (e.g.,  modify  priorities  for 
overlaying  targets). 

•  Reduce  size  of  ownship 

•  Include  ownship  velocity  vector 

•  Develop  a  simpler  means  for  de-cluttering  display. 

•  Include  target  selection  capability  and  display  associ¬ 
ated  information  in  all  traffic  modes. 

Crew  Workload 

Another  important  certification  issue,  crew 
workload,  was  assessed  using  the  Modified  Cooper- 
Harper  workload  scale,  and  by  including  several  ques¬ 
tions  on  the  flight  crew  questionnaire  related  to 
workload  (see  items  67,  68,  70,  89,  90,  94,  and  145  in 
Appendix  B).  The  Cooper  Harper  ratings  are  valuable 
in  that  they  partially  address  the  flight  crew  workload 
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certification  issue.  Comparisons  of  Cooper-Harper 
workload  ratings  and  the  overall  average  of  the  seven 
flight  crew  questionnaire  items  were  performed  to 
determine  the  extent  of  workload  for  two-  and  three- 
member  crews.  These  comparisons  were  dependent 
upon  and  included  ratings  from  all  CDTI  platform 
types.  The  Cooper-Harper  ratings  for  CDTI  plat¬ 
forms  indicated  that  three-member  crews  (ave  =  2.1) 
gave  significantly  lower  workload  ratings  than  two- 
member  crews  (ave  =  6.5).  A  rating  of  2.1  corresponds 
with  the  following  statement  on  the  Modified  Coo¬ 
per-Harper  Workload  Scale:  “Mental  effort  is  low  and 
desired  CDTI  performance  is  attainable.”  A  rating  of 
6.5  corresponds  with  “Maximum  mental  effort  is 
required  to  attain  acceptable  CDTI  performance.” 

Interpretation  of  the  workload  ratings  is  accompa¬ 
nied  by  several  caveats.  First,  although  flight  crews 
explicitly  mentioned  workload  in  the  context  of  the 
Final  Approach  Spacing  application,  flightdeck  ob¬ 
servers  suggested  that  the  PF  appeared  to  give  an 
overall  rating  for  all  of  the  spacing  tasks.  Hence, 
exactly  what  the  PF  rated  is  not  entirely  clear.  Second, 
the  workload  ratings  given  by  flight  crews  in  two-and 
three-person  crews  are  based  on  different  tasks  and 
different  platforms.  Furthermore,  the  flight  crews  in 
these  crews  had  differing  skill  levels.  Finally,  it  is 
important  to  remember  that  flight  crews  had  minimal 
training  on  the  CDTI  platform  (median  =  5  hrs)  and 
that  operating  environment  and  procedures  associ¬ 
ated  with  OpEval-2  were  not  the  same  as  those  expe¬ 
rienced  daily  by  line  flight  crews  in  normal  revenue 
operations.  Most  crewmembers  commented  that  the 
workload  associated  with  each  CDTI  platform  would 
be  manageable  with  more  experience,  better  proce¬ 
dures  design,  and  training. 

CDTI  Clutter 

The  analysis  of  OpEval-2  applications  provided  an 
opportunity  to  evaluate  the  CDTI  platform  display 
for  clutter.  The  flight  crew  questionnaire  contained 
four  items  about  clutter  (see  items  30,  31,  38,  50  in 
Appendix  B).  Although  flight  crews  rated  these  four 
items  favorably,  suggesting  that  display  clutter  was 
not  an  issue  for  the  CDTI  platform,  some  of  the  crews 
did  comment  about  display  clutter.  For  example,  they 
took  issue  with  the  momentary  merging  of  the  TTF 
target  with  a  grouping  of  surface  targets  as  TTF 
approached  the  runway.  This  phenomenon  compli¬ 
cated  the  FAROA  Air  task  for  the  aircraft  following, 
and  made  the  task  of  determining  runway  occupancy 
more  difficult.  Display  clutter  issues  appeared  to  be 
dependent  upon  the  type  of  OpEval-2  application;  in 
particular,  FAROA  and  ASSA.  Generally,  flight  crews 


wanted  a  more  effective  way  to  configure  the  CDTI 
platform  such  that  the  information  necessary  to  sup¬ 
port  the  task  was  displayed  and  that  which  was  unnec¬ 
essary  was  eliminated. 

Another  important  issue  associated  with  the  ASSA 
application  is  the  evaluation  of  surface  clutter  on  the 
traffic  display.  The  ASSA  application  portion  of  the 
flight  crew  questionnaire  included  two  statements 
(see  items  30  and  31  in  Appendix  B)  about  surface 
clutter  and  its  effect  on  flight  crews5  use  of  the  CDTI 
platforms  for  airport  surface  situation  awareness.  Al¬ 
though  a  majority  found  clutter  to  be  an  issue,  many 
did  not,  and  most  indicated  that  the  CDTI  de-clutter 
feature  was  effective  for  surface  applications.  How¬ 
ever,  post-flight  structured  interviews  revealed  that 
flight  crews  were  affected  by  surface  clutter  during  the 
FAROA  Air  task  as  previously  described.  This  clutter- 
related  phenomenon  should  be  mitigated. 

Display  Symbology 

Display  Symbology,  which  was  identified  as  a  us¬ 
ability  factor,  focused  on  color  coding  of  traffic  sym¬ 
bols,  consistency  of  color  and  symbology  usage  relative 
to  other  flightdeck  displays,  use  of  color  to  aid  deci¬ 
sion  making,  symbology  used  to  identify  ADS-B- 
equipped  traffic,  and  the  appropriateness  of  symbol 
sizes,  including  own-ship.  Overall,  the  ratings  for  the 
Display  Symbology  factor  were  favorable  for  the  CDTI 
platforms.  Although  flight  crews  rated  the  CDTI 
symbology  favorably,  the  symbology  for  depicting 
obstructions  on  the  airport  map  should  be  evaluated. 
No  symbology  for  obstructions  was  used  in  OpEval- 
2,  and  standard  symbology  is  needed  for  representing 
database  obstacles. 

Airport  Map  Attributes 

Airport  map  database  attributes  must  be  evaluated 
to  validate  the  accuracy,  resolution,  and  Integrity  of 
the  database.  Characteristics  of  Independently  de¬ 
signed  and  developed  databases  (i.e..  National  Oce¬ 
anic  and  Atmospheric  Agency,  Jeppesen,  and  others) 
should  be  compared,  and  chart  designs  and  their 
functionality  should  be  evaluated.  Although  the  map 
survey  group  at  NOAA  indicated  that  they  completed 
a  limited  validation  prior  to  its  use  at  OpEval-2,  the 
electronic  airport  map  database  was  not  validated 
completely.  Data  collected  during  OpEval-2  ASSA 
and  FAROA  applications  were  limited  to  measures 
associated  with  the  accuracy  of  aircraft  position  infor¬ 
mation  within  the  aircraft  map  database. 

Airport  map  database  critical  feature  attributes 
must  be  specified,  along  with  appropriate  levels  of 
accuracy,  resolution,  and  Integrity.  Attributes  that 
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may  be  supported  with  lesser  degrees  of  accuracy  also 
must  be  determined.  The  RTCA  Special  Committee- 
193  (SC- 193;  2001)  is  developing  standards  for  air¬ 
port  databases  and  has  suggested  three  levels  of  database 
accuracy:  coarse,  medium,  and  fine  (see  pp.  37  of  SC- 
193  standards  for  a  description  of  each  level).  Based 
on  taxi  task  requirements  for  navigation  and  geo¬ 
graphical  orientation  during  OpEval-2,  the  fine  reso¬ 
lution  database  was  selected  to  support  ASSA  and 
FAROA  tasks.  Although  no  specific  data  was  col¬ 
lected  to  determine  whether  the  “fine”  level  of  data¬ 
base  accuracy  was  required,  crew  responses  and 
performance  during  ASSA  and  FAROA  applications 
provides  some  data  pertaining  to  this  issue.  For  ex¬ 
ample,  during  surface,  final  approach,  landing,  and 
departure  operations,  flight  crews  reported  that  the 
CDTI  platform  generally  aided  both  airport  surface 
situation  and  traffic  awareness.  Additionally,  crews 
with  an  electronic  surface  map  reported  that  the 
system  accurately  depicted  their  position,  aided  in 
maintaining  geographical  orientation  during  taxi, 
aided  in  understanding  ATC  taxi  clearances,  and 
aided  in  determining  the  position  of  traffic  on  or  near 
runways  and  taxiways. 

The  level  of  accuracy  and  resolution  represented  by 
critical  attributes  in  the  airport  map  database  used  in 
OpEval-2  could  be  examined  and  evaluated  for  “com¬ 
pleteness”  by  RTCA  SC- 193.  The  surface  vehicle 
used  to  test  the  airport  map  database  traversed  many 
of  the  Standiford  International  Airport  taxiways  prior 
to  OpEval-2  and  personnel  who  performed  this  test¬ 
ing  did  not  report  any  “incomplete”  features.  Further¬ 
more,  flight  crews  who  participated  in  OpEval-2 
indicated  that  the  display  of  critical  airport  map 
features  on  the  CDTI  platform  corresponded  with 
what  they  observed  on  the  airport  surface.  These 
findings  are  tempered  because  both  flight  crews  and 
ground  survey  personnel  were  unfamiliar  with  criteria 
for  determining  which  features  were  critical  to  surface 
navigation  and  airport  surface  situation  awareness. 
Research  on  airport  surface  operations  has  not  pro¬ 
vided  sufficient  evidence  to  support  identification  of 


features  critical  to  surface  navigation  and  airport 
surface  situation  awareness.  In  an  attempt  to  gather 
more  information  about  critical  features  that  may 
have  influenced  flight  crew  performance  during 
OpEval-2  ASSA  applications,  day  and  night  vehicle 
surveys  of  four  basic  taxi  routes  were  completed 
immediately  after  OpEval-2.  Initial  results  of  these 
surveys  led  to  the  identification  of  some  locations  on 
the  airport  where  airport  signage  and  taxiway  lines 
were  either  missing  or  confusing.  Analyses  of  the 
OpEval-2  data  from  the  ASSA  application  revealed 
that  taxi  errors  were  associated  with  airport  surface 
locations  where  information  was  missing.  Finally, 
additional  research  is  needed  to  determine  which 
airport  and  database  features  are  critical  to  surface 
navigation  and  airport  surface  situation  awareness, 
and  to  develop  a  hierarchy  for  including  these  features 
in  electronic  airport  map  databases. 
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APPENDIX  A 


ACRONYMS 


ADS-B 

Automatic  Dependent  Surveillance  -  Broadcast 

ANOVA 

Analysis  of  Variance 

ARTCC 

Air  Route  Traffic  Control  Center 

ASSA 

Airport  Surface  Situation  Awareness 

ATC 

Air  Traffic  Control 

CAA 

Cargo  Airline  Association 

CAASD 

Center  for  Advanced  Aviation  System  Development 

CDTI 

Cockpit  Display  of  Traffic  Information 

FAA 

Federal  Aviation  Administration 

FAF 

Final  Approach  Fix 

FAROA 

Final  Approach  and  Runway  Occupancy  Awareness 

GPS 

Global  Positioning  System 

HUD 

Head-up  Display 

I-Lab 

Integration  and  Interaction  Laboratory 

LDPU 

Link  and  Display  Processor  Unit 

NAS 

National  Airspace  System 

NM 

Nautical  Miles 

NOTAM 

Notice  to  Airmen 

OCG 

Operational  Evaluation  Coordination  Group 

OpEval 

Operational  Evaluation 

PF 

Pilot  Flying 

PNF 

Pilot  Not  Flying 

SDF 

Louisville  International  Airport 

SF21 

SafeFlight  21 

TCAS 

Traffic  Alert  and  Collision  Avoidance  System 

TEMP 

Test  and  Evaluation  Master  Plan 

TIS 

Traffic  Information  Service 

TIS-B 

Traffic  Information  Service-Broadcast 

TRACON 

Terminal  Radar  Control 

TSO 

Technical  Standard  Order 

TTF 

Traffic  to  Follow 

UAT 

Universal  Access  Transceiver 

UPS  AT 

UPS  Aviation  Technologies 
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APPENDIX  B:  FLIGHTDECK  OBSERVER  DATA-COLLECTION  TOOLS 

CDTI  BUILD  2/4  AND  PROTOTYPE  PLATFORMS  FLIGHT  CREW  QUESTIONNAIRE 

See  Adobe  Acrobat  file  (CDTI  Questionnaire.pdf) 

MX-20  PLATFORM  FLIGHT  CREW  QUESTIONNAIRE 

The  purpose  of  this  questionnaire  is  to  evaluate  flight  crew  human  factors  related  to  the  use  of  the  MX-20  for  the 
applications  evaluated  m  the  second  Safe  Flight  21  Operational  Evaluation  (OpEval-2).  Results  of  this  questionnaire 
and  data  collected  during  the  OpEval-2  may  be  published;  however,  your  identity  will  be  kept  confidential  in 
accordance  with  45  CFR  Part  46  of  1981,  and  49  CFR  Part  1 1  of  19915.  Neither  your  employer  nor  any  regulatory 

agency  shall  have  access  to  your  identity,  as  it  appears  in  any  of  the  data  collected  from  this  questionnaire  or  other 
sources  for  the  purposes  of  this  study. 

Please  use  the  following  point  of  contact  if  you  have  questions  about  the  data  collection  procedures  or  have  related 
concerns, 

B,  Oscar  Olmos 
The  MITRE  Corporation 

Center  for  Advanced  Aviation  System  Development 
1 S20  Dolley  Madison  Blvd 
McLean,  VA  22102-3481 
(703)  883-5746 


Date:  October  | _ j _  ,2000 

Example  2  5 

Flight  Event:  | _ | 

Example  3 
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CDTI  BUILD  2/4  AND  PROTOTYPE  PLATFORMS  FLIGHT  CREW  QUESTIONNAIRE 

Pilot  Name: _ _ _ _ 

Aircraft  Tail  Number:  _ _ _ — 

Crew  Position  (please  indicate  your  response  by  placing  a  "X"  on  the  appropriate  line): 

_ Captain  _ First  Officer 

Flight  Crew  Information 

1.  FAA  Certificates/Ratings  (please  indicate  your  response  by  placing  a  "X"  on  ALL  applicable  lines): 
_ Private 

_ Commercial 

_ ATP 

_ Instrument 

_ CFI 

_ CHI 

_ Other  (please  write  in  your  response  -»): - - - 

2.  Aircraft  Type  Ratings  (please  write  in  your  response): 


Estimate  the  number  of  hours  flown. 

3.  Total  Hours 

Flown:  | _ | _ | _ | _ 

4.  Total  VFR:  I  1  I  1  1 

5.  Total  IFR:  1  1  I  I  I 

6.  Last  12  months:  | _ | _ 1 _ ] _ | 

7.  Last  90  days:  | _ | _ | _ | 


Operational  Experience  (please  estimate  the  percentage  of  time  that  you  fly  under  each  FAR  part): 
Items  8-11  should  total  100%. 


8. 

9. 

10. 

11. 


Part 

121: 

Part 

135: 

Part  91: 

Other: 


% 

% 

1111  * 
1111  % 


B2 


Previous  MX-20/GX-60  Experience 

12.  Have  you  flown  MX-20/GX-60  equipped  aircraft  before  this  date? 

_ No 

- Yes  If  so,  please  answer  the  following  by  placing  a  "X"  on  ALL  applicable  lines. 

_ as  a  test  flight 

_ as  part  of  revenue  service 

- other  (please  write  in  your  response  — »); 

13*  P^ease  indicate  the  total  number  of  hours  flown  using  the  MX-2G/GX- 

Example  2  5  0 

‘lease  indicate  any  MX-20  training  you  have  had  by  putting  an  “X”  in  front  of  the  type  of  training. 
_ OpEval-2  training 

A)  Was  this  training  helpful? _ Not  helpful _ Somewhat  helpful _ Very  helpful. 

B)  Please  indicate  the  number  of  training  hours. 

_ Formal  training  (e.g.,  course) 

A)  Was  this  training  helpful? _ Not  helpful _ Somewhat  helpful _ Very  helpful. 

B)  Please  indicate  the  number  of  training  hours. _ 

- Informal  training  (e.g.,  reading  manual) 

A)  Was  this  training  helpful? _ Not  helpful _ Somewhat  helpful _ Very  helpful. 

B)  Please  indicate  the  number  of  training  hours. _ 

- Other  (please  write  in  your  response): _ 

A)  Was  this  training  helpful? - Not  helpful _ Somewhat  helpful _ Very  helpful. 

B)  Please  indicate  the  number  of  training  hours. _ _ 
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Each  of  the  following  statements  should  be  answered  in  relation  to  various  aspects  of  both  the  GPS  unit  (GX- 
60)  and  the  multi-function  display  unit  (MX-20).  Use  the  following  scale  to  indicate  your  level  of  agreement 
with  each  statement. 

0  =  Not  Applicable/Did  Not  Use 

1  =  Strongly  Disagree 

2  =  Somewhat  Disagree 

3  =  Neither  Disagree  nor  Agree 

4  =  Somewhat  Agree 

5  =  Strongly  Agree 


GX-60 


Statement 

Startup 

Creating 
Flight  Plan 

Editing 
Flight  Plan 

En  Route 
Navigation 

Responding 
to  System 
Messages 

Accessing 

Airport 

Information 

15. 

The  equipment  is  easy  to  use. 

16. 

The  equipment  operating 
procedures  are  easy  to  remember. 

17. 

The  equipment  performs  the 
functions  necessary  for  my  flying 
operations. 

18. 

The  equipment  provides  most  of 
the  information  I  need  to  conduct 
my  flight. 

19. 

The  equipment  allows  me  to 
perform  all  necessary  flight 
functions  without  reference  to 
operating  manuals,  etc. 

20. 

The  equipment  provides  easy 
access  to  the  functions  necessary 
for  my  flight. 

i 

21. 

The  equipment  operating  manual 
clearly  explains  procedures. 

22. 

When  I  press  the  wrong  button,  it  is 
easy  to  undo. 

23. 

I  am  never  confused  about  which 
display  page  is  active. 

24. 

The  equipment  accommodates 
"canned"  flight  plans. 

25. 

The  menu  choices  or  button  formats 
are  desirable. 

26. 

The  menus  are  easy  to  find. 
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Statement 


27. 1  The  equipment  is  easy  to  use. 


The  equipment  operating 
procedures  are  easy  to  remember. 


The  equipment  performs  the 
functions  necessary  for  my  flying 
operations. 


The  equipment  provides  most  of 
the  information  I  need  to  conduct 

my  flight.  _ _ 

The  equipment  allows  me  to 
perform  all  necessary  flight 
functions  without  reference  to 
operating  manuals,  etc. 


The  equipment  provides  easy 
32.  access  to  the  functions  necessary 
for  my  flight. 


The  equipment  operating  manual 
clearly  explains  procedures. 


When  I  press  the  wrong  button,  it 

is  easy  to  undo. _ 

I  am  never  confused  about  which 
display  page  is  active. 


Accessing 
and  Using 
Custom  Map 
Function 


MX-20 

Accessing 
and  Using 
Terrain 


Accessing 
and  Using 
Traffic 
Function 


Accessing 
and  Using 
Flight  Plan 
Function 


Accessing 
and  Using 
VFR  Chart 
Function 


Accessing 
and  Using 
IFR  Chart 
Function 


36. 

The  PAN  feature  is  desirable  and 
easy  to  use. 

37. 

The  INFO  mode  is  very  useful. 

38, 

The  North-Up  mode  is  preferred. 

39. 

The  Track-Up  mode  is  preferred. 

40. 

It's  easy  to  adjust  the  map  scale. 

41. 

The  available  map  scales  were 
appropriate  to  my  flight  tasks. 

42. 

The  advisory  flags  (terrain  or 
traffic)  are  easy  to  respond  to. 

43. 

The  360  mode  is  preferred. 

44. 

The  Arc  mode  is  preferred. 

45. 

The  INVERT  function  is 
desirable. 

46. 

The  NAV  data  function  is 
desirable. 

Statement 


49.  The  response  time  of  the  equipment  is  adequate. 


50.  The  equipment  provides  adequate  feedback. 


51.  The  display  does  not  wash  out  in  direct  sunlight. 


The  display  does  not  wash  out  in  indirect  sunlight. 


The  display  is  legible  in  night  conditions. 


54.  The  controls  are  easy  to  operate. 


55.  Control  labels  are  easy  to  understand  and  remember. _ 

56.  The  equipment  allows  for  easy  detection  of  alerting  messages. _ 

57.  Alerting  messages  are  appropriate  and  easily  understood. _ 

58.  The  equipment-operating  manual  is  easy  to  use. 


59.  I  feel  confident  using  the  equipment  for  VFR  navigation. 


6Q  I  feel  confident  using  the  equipment  to  aid  in  the  visual  acquisition  of 

_  other  aircraft. _ _ 

.  I  feel  confident  using  the  equipment  to  aid  in  separation  from  terrain 

_  during  VFR  flight. _ _ 

62.  The  equipment  is  helpful  as  a  supplemental  system  during  IFR  flight. 


^  The  colors  used  to  code  the  traffic  symbols  on  the  display  were 
appropriate. 


6^  The  color  coding  used  on  the  display  was  consistent  with  that  of  other 

_  flight  deck  displays. _ 

65.  The  colors  used  on  the  display  aided  decision-making. 


66  The  symbology  used  on  the  display  was  consistent  with  that  of  other 
flight  deck  displays. 


67.  The  scale  of  the  objects  on  the  display  was  appropriate. 


I  was  distracted  from  other  flying  tasks  when  using  the  equipment. 


Overall,  use  of  the  equipment  enhanced  my  decision-making. _ 

Overall,  use  of  the  equipment  did  not  increase  head-down  time. _ 

Overall,  use  of  the  equipment  did  not  increase  workload. 


72.  Overall,  the  equipment  contributed  to  the  successful  completion  of  tasks. 


73.  The  physical  layout  of  buttons  allowed  them  to  be  used  accurately. 


74.  The  size  of  the  buttons  made  them  easy  to  use. _ 

75.  Button  presses  were  accompanied  by  appropriate  feedback. _ 

76.  The  location  of  the  equipment  allowed  it  to  be  seen  easily. _ 

^  The  reach  required  to  use  the  equipment  was  acceptable  from  my  seating 

position. 


The  readability  of  the  text  on  the  equipment  display  was  acceptable  from 

_  my  seating  position. _ 

^  Having  to  read  the  text  displays  interfered  with  the  performance  of  flight 
duties. 


gQ  The  addition  of  the  equipment  in  the  cockpit  decreased  the  time 
available  to  scan  instruments. 
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Airport  Surface  Situation  Awareness 


Statement 

Not 

applicable/ 
Did  not  use 

Strongly 

disagree 

Somewhat 

disagree 

Neither 
disagree 
nor  agree 

Somewhat 

agree 

Overall,  use  of  the  MX-20  increased  my  ability 
to  accurately  locate  other  aircraft  on  the 
ground. 

o 

0 

o 

o 

0 

Use  of  the  MX-20  for  airport  surface  situation 
awareness  aided  my  ability  to  understand  ATC 
communications. 

o 

o 

o 

o 

0 

The  relative  position  of  other  traffic  was  easy 
to  interpret  and  understand  on  the  MX-20. 

o 

o 

o 

o 

o 

It  was  easy  to  get  oriented  to  the  surface  map 
display. 

o 

o 

o 

o 

o 

In  general,  enough  information  was  provided 
on  the  MX-20  to  be  useful. 

o 

o 

o 

o 

o 

In  general,  more  information  should  be 
displayed  on  the  MX-20  to  aid  in  airport 
surface  situation  awareness. 

o 

o 

o 

o 

° 

The  MX-20  display  accurately  showed  my 
position  on  the  ground  (taxiway). 


identify 


o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

If  you  are  in  disagreement  with  item  89,  please  explain  why? 


o  i  The  surface  map  was  easy  to  bring  up  on  the  T 

_  display. _ _  O  O  O  O  O  O 

92  Use  of  the  MX-20  during  taxi  increased  the  _  " 

time  available  for  crew  duties,  ^  O  O  O  O  O 

^3  Use  of  the  MX-20  during  taxi  increased  the  T~  ” 

_  time  available  for  completing  checklists.  ®  ®  ®  O  O  O 

94  Using  MX-20  on  the  airport  surface  aided  in  T 

locating  targets  visually. _  ^  O  O  O  O  O 

)5  Use  of  the  MX-20  aided  in  supporting  traffic  “ 

awareness  while  on  the  airport  surface. _  °  O  O  O  O  O 

^  Display  clutter  was  not  a  problem  during  T 

airport  surface  operations. _  ®  O  O  O  O  O 

Use  of  the  MX-20  during  airport  surface  ~ 

_  operations  did  not  increase  head-down  time.  °  °  °  °  °  ° 

If  in  agreement  with  item  97,  did  using  the  MX-20  enhance  or  reduce  the  safety  of  airport  surface  operations? 


Do  you  feel  the  MX-20  helped,  hindered,  or  had  no  effect  on  your  overall  performance  during  departure 
gg  spacing  operations? _ _ 


Departure  Spacing 


riease  muiuaic  yuui  icvci  iv  mv  *vuv»M,.b 

Statement 

Not 

applicable/ 
Did  not  use 

- D - - 

Strongly 

disagree 

Somewhat 

disagree 

Neither 
disagree 
nor  agree 

Somewhat 

agree 

Strongly 

agree 

100. 

Use  of  the  MX-20  for  departure/climb-out 
aided  in  traffic  awareness. 

o 

o 

o 

o 

0 

o 

101. 

The  MX-20  range  settings  were  adjusted  to 
increase  traffic  awareness  during  departure. 

o 

o 

o 

o 

o 

o 

102. 

Display  clutter  was  not  a  problem  during 
departure  spacing  operations. 

o 

o 

o 

o 

0 

0 

103. 

Use  of  the  MX-20  during  departure  spacing 
operations  did  not  increase  head-down  time. 

o 

o 

o 

o 

0 

o 

104. 

If  in  agreement  with  item  103,  did  using  the  MX-20  enhance  or  reduce  the  safety  of  departure  spacing 

105. 

It  was  easy  to  transition  from  final  approach  to 
surface  operations  using  the  MX-20. 

0 

o 

o  . 

o 

o 

0 

106. 

The  procedure  for  using  the  MX-20  to 
transition  from  final  approach  to  surface 
nnerations  was  acceptable. 

o 

o 

o 

o 

o 

0 

107. 

Do  you  feel  the  MX-20  helped,  hindered,  or  had  no  effect  on  your  overall  performance  during  departure 

Visual  Acquisition 


Please  indicate  your  level  ot  agreement  to  tne  iouowing 

Statement 

□y  oiacKci 

Not 

applicable/ 
Did  not  use 

Ull£  LUC  C1J 

Strongly 

disagree 

C1C  111 

Somewhat 

disagree 

Neither 
disagree 
nor  agree 

Somewhat 

agree 

Strongly 

agree 

108. 

Use  of  the  MX-20  aided  in  visually  acquiring 
traffic  before  receiving  an  ATC  call. 

o 

o 

o 

o 

o 

o 

109. 

Use  of  the  MX-20  aided  in  visually  acquiring 
traffic  after  receiving  an  ATC  call. 

0 

o 

o 

o 

0 

o 

110. 

ATC  traffic,  when  visually  acquired,  appeared 
at  the  same  clock  position  as  depicted  on  the 
MX-20. 

o 

o 

o 

o 

0 

o 

111. 

ATC  traffic,  when  visually  acquired,  appeared 
at  the  same  distance  as  depicted  on  the  MX-20. 

0 

0 

o 

o 

0 

o 

112. 

Use  of  the  MX-20  had  no  effect  on  maintaining 
awareness  of  multiple  traffic  targets. 

0 

o 

o 

o 

o 

o 

113. 

Display  clutter  was  not  a  problem  when  trying 
to  acquire  aircraft  visually  using  the  MX-20. 

o 

o 

o 

o 

o 

o 

114. 

The  Select  function  was  useful  for  visually 
acquiring  traffic. 

o 

o 

o 

o 

o 

o 

115. 

Use  of  the  MX-20  to  enhance  visual  acquisition 
did  not  increase  head-down  time. 

o 

o 

o 

o 

o 

o 

116. 

If  in  agreement  with  item  1 16,  did  using  the  MX-20  enhance  or  reduce  the  safety  of  visual  acquisition 

117. 

When  using  the  MX-20,  were  you  distracted  when  other  pilots  used  your  call  sign  /  It  so,  please  explain. 
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Final  Approach  Runway  Occupancy  Awareness 

_  .  «  «  p  ,  *i  r.n _ 


Statement 

Not 

applicable/ 
Did  not  use 

Strongly 

disagree 

Somewhat 

disagree 

Neither 
disagree 
nor  agree 

Somewhat 

agree 

Strongly 

agree 

136. 

Use  of  the  MX-20  effectively  enhanced  my 
awareness  of  final  approach  traffic. 

o 

0 

o 

o 

0 

0 

137. 

Use  of  the  MX-20  effectively  enhanced  my 
awareness  of  departure  traffic. 

o 

o 

o 

o 

0 

o 

138. 

Observing  the  target  change  color  (e.g.,  brown 
to  cyan)  was  helpful  in  determining  the  flight 
status  of  the  target. 

o 

o 

0 

o 

0 

0 

139. 

When  in  take-off  position,  use  of  the  MX-20 
enhanced  my  awareness  of  aircraft  clearing  the 
runway. 

o 

o 

o 

o 

o 

0 

140. 

When  in  take-off  position,  use  of  the  MX-20 
enhanced  my  awareness  of  ground  aircraft  and 
vehicles  on  nearby  taxi  ways. 

o 

0 

o 

o 

0 

0 

141. 

The  MX-20  aided  in  gauging  when  the  traffic 
to  follow  was  over  the  runway  threshold. 

o 

0 

o 

o 

o 

o 

142. 

The  MX-20  aided  in  gauging  when  the  traffic 
to  follow  was  touching  down. 

o 

o 

o 

o 

o 

0 

143. 

The  MX-20  aided  in  gauging  when  the  traffic 
to  follow  was  clear  of  the  runway. 

o 

o 

o 

o 

o 

0 

144. 

During  low  approaches,  the  MX-20  aided  in 
detecting  targets  positioned  on  the  runway. 

o 

0 

o 

o 

o 

o 

145. 

During  low  approaches,  the  MX-20  aided  in 
detecting  targets  positioned  near  the  runway. 

o 

o 

o 

o 

o 

o 

146. 

The  MX-20  aided  in  detecting  targets  crossing 
the  runway. 

o 

o 

o 

o 

o 

o 

Please  indicate  your  response  to  the  following  statements  by  using  the  scale  below. 

1  =  Poor 

2  =  Fair 

3  =  Good 

4  =  Very  good 

5  =  Excellent 

How  would  you  rate  the  MX-20  overall  as  an  aid  to. . . 

147.  . .  .airport  surface  situation  awareness? _ 

148.  ...  departure  spacing?  _ _ 

149.  . .  .visual  acquisition? _ _ _ 

150.  . .  .approach  spacing? _ 


BIO 


Comments/Suggestions 


Each  of  the  following  statements  should  be  answered  in  relation  to  various  aspects  of  both 
the  GPS  unit  (GX-60)  and  the  multi-function  display  unit  (MX-20).  Comments  will  be 
summarized,  so  please  print  or  write  clearly  and  be  concise. 

151.  Do  you  have  any  comments  regarding  the  use  of  color  and  symbology  on  the  GX-60? 


152.  Do  you  have  any  comments  regarding  the  use  of  color  and  symbology  on  the  MX-20? 


153,  Do  you  have  any  additional  comments  regarding  specific  problems  you  have  encountered 
with  the  GX-60? 


154.  Do  you  have  any  additional  comments  regarding  specific  problems  you  have  encountered 
with  the  MX-20? 


155.  What  recommendations  would  you  make  for  improving  the  design  of  the  GX-60? 


15'6.  What  recommendations  would  you  make  for  improving  the  design  of  the  MX-20? 

157.  What  recommendations  would  you  make  for  improving  the  use  of  the  GX-60? 

158.  What  recommendations  would  you  make  for  improving  the  use  of  the  MX-20? 


This  information  is  NOT  required  but  would  be  appreciated.  If  we  require  clarification  of  any  of 
your  responses,  may  we  contact  you?  If  your  answer  is  yes,  please  provide  the  following 
information. 


Phone:  ( _ ) 

E-mail: _ 


Thank  you  for  your  participation! 
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CHECKLISTS  FORM 


Flight  Event:  ©  ©  (D  ©  © _  Aircraft  Tail  Number:  N _ Date:  10/ _ /00 

Circuit:  ©  ©  ®  ©  ©  ©  Page:  ©  ©  Flying  Pilot:  O  Captain  O  FO 


G  =  Ground  U  =  Upwind  X  =  Crosswind  D  =  Downwind  B  =  Base  F  =  Final 


Checklist  Name  (1) 

Status 

Discuss  CDTI  Effects  ONLY 

O  Completed  O  Missed  by  Crew 

O  Not  Completed  O  Missed  by  Observer 

O  Interrupted  O  Other 

O  Suspension 

Where  completed  (circle  one): 
G  U  X  D  B  F 

Checklist  Name  (2) 

Status 

Discuss  CDTI  Effects  ONLY 

O  Completed  O  Missed  by  Crew 

0  Not  Completed  O  Missed  by  Observer 

O  Interrupted  O  Other 

O  Suspension 

Where  completed  (circle  one): 
G  U  X  D  B  F 

Checklist  Name  (3) 

Status 

Discuss  CDTI  Effects  ONLY 

O  Completed  O  Missed  by  Crew 

O  Not  Completed  O  Missed  by  Observer 

O  Interrupted  O  Other 

O  Suspension 

Where  completed  (circle  one): 
G  U  X  D  B  F 

Checklist  Name  (4) 

Status 

Discuss  CDTI  Effects  ONLY 

O  Completed  O  Missed  by  Crew 

O  Not  Completed  O  Missed  by  Observer 

O  Interrupted  O  Other 

O  Suspension 

Where  completed  (circle  one): 
G  U  X  D  B  F 

Checklist  Name  (5) 

Status 

Discuss  CDTI  Effects  ONLY 

_  ^  O  Missed  by  Crew 

O  Completed  J 

_  _  _  ,  ,  O  Missed  by  Observer 

O  Not  Completed  J 

*  T  ,  O  Other 

O  Interrupted 

O  Suspension 

Where  completed  (circle  one): 
G  U  X  D  B  F 

Checklist  Name  (6) 

Status 

Discuss  CDTI  Effects  ONLY 

O  Completed  O  Missed  by  Crew 

O  Not  Completed  O  Missed  by  Observer 

O  Interrupted  O  Other 

O  Suspension 

Where  completed  (circle  one): 
G  U  X  D  B  F 
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VISUAL  ACQUISITION  AND  TRAFFIC  AWARENESS  FORM 


Flight  Event:©  ©  ©  ®  © _ Aircraft  Tail  Number:  N _ 

Date:  10/ _ /0O  Circuit:  ©  @  ©  ©  ®  ©  Flying  Pilot:  O  Captain  O  FO 


Traffic  1 

Traffic  2 

Traffic  3 

Traffic  4 

Traffic  5 

Traffic6 

SaiMi 

faiMn 

Aircraft 

ID/Type 

HHfcJTgEjJggM 

/ 

/ 

/ 

/ 

/ 

- 

- 

- 

_ 

Order "  V  * 

Order 

Order 

PPjKOrder 

%C: ;  x  Order 

Visual  ©  ©  ® 

Visual  ©  ©  © 

Visual  ©  ©  © 

Visual  ©  ©  © 

Visual  ©  ©  © 

Visual  ©  ©  © 

CDTI  ©  ©  © 

CDTI  ©  ©  © 

CDTI  ©  ©  © 

CDTI  ©  ©  © 

CDTI  ©  ©  © 

CDTI  ©@® 

ATC  ©  ©  © 

ATC  ©  ©  ® 

ATC  ©  ©  © 

ATC  ©  ©  © 

ATC  ©  ©  © 

ATC  ©  ©  © 

Crew/ATC 

Queries 

Crew/ATC 

Queries 

©  ©  ©  ©  © 

©  ©  ©  ©  © 

HjHHNj 

©  ®  ©  ©  © 

1 

?  1  ^  Notes 

,  if-  Notes 

t 1  i  Notes  r? 

Notes 

Notes 

Notes 

Call  sign  used  ? 

Call  sign  used  ? 

Call  sign  used  ? 

Call  sign  used  ? 

Call  sign  used  ? 

Call  sign  used  ? 

O  Yes  O  No 

O  Yes  O  No 

O  Yes  O  No 

O  Yes  O  No 

O  Yes  O  No 

O  Yes  O  No 

Traffic  7 

Traffic  8 

Traffic  9 

Traffic  10 

Traffic  11 

Traffic  12 

I:®:  Aira*aft",®| 
ID/Type  •  /:§§ 

H  ^Alrfiraft 
ID/Type 

ID/Type 

/ 

/ 

/ 

/ 

/ 

/ 

'  •;#  Order  -  ;< 

Order 

Order 

Order 

Orders 

Visual  ©  ©  © 

Visual  ©  ©  © 

Visual  ©  @  © 

Visual  ©  ©  ® 

Visual  ©  ©  © 

Visual  ©  ©  © 

CDTI  ©  ©  © 

CDTI  ©  ©  © 

CDTI  ©  ©  © 

CDTI  ©  ©  © 

CDTI  ©  ©  ® 

CDTI  ©  ©  © 

ATC  ©  ©  © 

ATC  ©  ©  © 

ATC  ©  ©  © 

ATC  ©  ©  © 

ATC  ©  ©  © 

ATC  ©  ©  © 

Crew/ATC  a 
Queries  ■ 

Crew/ATC 

Queries 

Crew/ATC  . 
Queries 

Crew/ATC 

Queries 

Crew/ATC 

Queries 

Crew/ATC 

Queries 

©  ©  ©  ©  © 

©  ©  ©  ©  © 

©  ©  ©  ©  © 

©  ©  ®  ©  © 

©  ©  ®  ©  © 

- JSb - 1 _ 

©  ©  ©  ©  © 

Notes 

Notes 

Notes 

'■  -  :v. 

Notes  - 

Notes 

Call  sign  used  ? 

Call  sign  used  ? 

Call  sign  used  ? 

Call  sign  used  ? 

Call  sign  used  ? 

Call  sign  used  ? 

0  Yes  O  No 

O  Yes  O  No 

O  Yes  O  No 

O  Yes  O  No 

O  Yes  O  No 

O  Yes  O  No 
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MX-20  PLATFORM  PROCEDURES  FORM 


Flight  Event:  (D  ©  (D  ®  ®  _  Aircraft  Tail  Number:  N 

Date:  10/ _ /00  Flying  Pilot:  O  Captain  O  FO 


Airport  Surface  Situation  Awareness  (ASSA) 

CDTI  Used  to... 

Cl 

C2 

C3 

C4 

Notes 

— 

■f 

■y 

Y 

1.  ...determine  taxi  sequence? 

■a 

HI 

mm 

N 

2.  ...evaluate  local  traffic  situations? 

Y 

N 

Y 

N 

H 

Y 

N 

Y 

Y 

n 

Y 

3.  ...locate  targets  relative  to  ground 

N 

N 

II 

N 

reference  points  (e.g.,  taxiways)? 

N/A 

N/A 

BB 

N/A 

CDTI  Features 

Cl 

C2 

■SB 

C4 

Notes 

1.  Use  Select  to  identify  assigned  taxi 

Y 

Y 

mm 

Y 

interval? 

N 

N 

N 

2.  Use  Time  (Vector  lines)  to  evaluate 

Y 

Y 

mu 

Y 

target  speed? 

N 

N 

mm 

N 

Final  Approach  Runway  Occupancy  Awareness  i 

[FARO/ 

0  while  on  GROUND 

CDTI  Used  to... 

Cl 

C2 

C3 

C4 

Notes 

— 

Y 

1.  ...evaluate  runway  occupancy? 

mm 

N 

2.  ...evaluate  final  approach  corridor  for 

’  Y 

S3 

mm 

mm 

possible  conflicts? 

N 

mSm 

mm 

3.  ..  .evaluate  departure  corridor  for 

Y 

Y 

Y 

mm 

possible  conflicts? 

N 

N 

N 

CDTI  Features 

Cl 

C2 

C3 

MSM 

Notes 

N-Up 

N-Up 

N-Up 

N-Up 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

1 .  Which  map  orientation  was  used  most 

Arc 

Arc 

Arc 

Arc 

frequently? 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

360 

360 

360 

Desired 

Desired 

Desired 

Desired 

Trk-Up 

Trk-Up 

Trk-Up 

EE5S3 

2.  Use  In  and  Out  to  adjust  range  during 

Y 

Y 

Y 

mm 

taxi? 

N 

N 

N 

3.  Use  Time  (Vector  lines)  to  evaluate 

Y 

Y 

Y 

mm 

target  speed? 

N 

N 

N 

n 

CDTI  Used  to... 

Cl 

C2 

C3 

C4 

Notes 

Y 

Y 

Y 

Y 

1.  ...determine spacing? 

N 

N 

N 

N 

CDTI  Features 

Cl 

C2 

C3 

C4 

Notes 

1.  Use  In  and  Out  to  adjust  range  prior 

Y 

Y 

Y 

Y 

to  takeoff  roll? 

N 

N 

N 

N 

2.  Use  Traffic  Altitude  Filter  to  set 

Y 

Y 

Y 

Y 

altitude  range  filter  for  departure 

N 

N 

N 

N 

spacing? 
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Initial  Approach  Spacing  (Crosswind,  Downwind  and  Base  Less) 

CDTI  Used  to... 

Cl 

C2 

C3 

C4 

Notes 

L  ...establish  and  maintain  spacing? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

CDTI  Features 

Cl 

C2 

C3 

C4 

Notes 

1 .  Use  Select  to  identify  target? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2.  Use  Time  (Vector  lines)  to  aid  in 
spacing? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

3.  Use  Traffic  Altitude  Filter  to  set 
altitude  range  filter  for  initial  approach 
spacing? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

4.  Use  Time  (Vector  lines)  to  evaluate 
target  speed? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Final  Approach  Spacing 

CDTI  Used  to..* 

'Cl  St 

C2  5 

C3 

C4 

•••h Notes'  '  ; ‘ 

1.  ..  .establish  and  maintain  spacing? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2.  , .  .evaluate  spacing  by  pilot  not  flying 

Y 

Y 

Y 

Y 

ONLY? 

N 

N 

N 

N 

CDTI  Features 

Cl 

C2 

C3 

C4 

Notes 

1.  Use  Select  to  identify  target? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2.  Set  Time  (Vector  lines)  to  aid  in 

Y 

Y 

Y 

Y 

spacing? 

N 

N 

N 

N 

3.  Use  Time  (Vector  lines)  to  evaluate 

Y 

Y 

Y 

Y 

target  speed? 

N 

N 

N 

N 
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Final  ADDroach  Runway  Occupancy  Awareness  ( 

FAROA)  while  in  AIR 

CDTI  Used  to... 

Cl 

C2 

C3 

C4 

Notes 

1.  ...evaluate  the  runway  environment 

Y 

Y 

Y 

Y 

for  possible  conflicts? 

N 

N 

N 

N 

CDTI  Features 

Cl 

C2 

C3 

C4 

Notes 

1.  Use  In  and  Out  to  adjust  range  within 

Y 

Y 

Y 

Y 

runway  environment? 

N 

N 

N 

N 

N-Up 

N-Up 

N-Up 

N-Up 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

2.  Which  map  orientation  was  used  most 

Arc 

Arc 

Arc 

Arc 

frequently? 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

360 

360 

360 

360 

Desired 

Desired 

Desired 

Desired 

Trk-Up 

Trk-Up 

Trk-Up 

Trk-Up 

3.  Use  Time  (Vector  lines)  to  evaluate 

Y 

Y 

Y 

Y 

target  speed? 

N 

N 

N 

N 

Runway  Incursion  Scenarios  (Flight  Periods  1  and  2  ONLY) 

Notes 

1.  Was  TRIOSl  Van  identified?  CDTI  ©@  Visual  ©@  No  O 

Distance  nm  (check  CDTI  for  distance) 

2.  Was  FAA  727  (N40)  identified?  CDTI  ©  ©  Visual  ©  ©  No  O 

Distance  nm  (check  CDTI  for  distance) 
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Surface  Navigation  | 

CDTI  Used  to... 

Cl 

C2 

C4 

Notes 

1.  ...evaluate  the  airport  surface 
environment  for  possible  conflicts? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2. . .  .identify  taxiways  on  airport 
surface? 

Y 

N 

Y 

N 

B 

CDTI  Features 

Cl 

,#C2 

J:;':C31, 

Notes 

1.  Use  In  and  Out  to  adjust  range  on 
airport  surface? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

2.  Which  map  orientation  was  used  most 
frequently? 

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up 

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up 

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up 

N-Up 

Trk-Up 

Trk-Up 

Arc 

Trk-Up 

360 

Desired 

Trk-Up 

3.  Use  Time  (Vector  lines)  to  evaluate 
target  speed? 

Y 

N 

Y 

N 

Y 

N 

B 

4.  Use  Select  to  identify  target? 

Y 

N 

Y 

N 

H 

O  Never  O  Sometimes 

5.  The  Surface  Map  was  used  to  taxi... 

O  Always  O  N/A 

(If  Available) 

Location? 

Proximate  AC  ID? 

Outbound 

O  Structured 

O  Unstructured 

Start 

O  Correct 

O  Incorrect 

AC  Type 

Taxi  Speed 

Stop 

ACID 

Inbound 

O  Structured 

O  Unstructured 

Start 

O  Correct 

O  Incorrect 

AC  Type 

Taxi  Speed 

Stop 

ACID 
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CDTI  BUILD  2/4  AND  PROTOTYPE  PROCEDURES  FORMS 


Flight  Event:  ©  ©  ®  ©  ® _  Aircraft  Tail  Number:  N 

Date:  10/ _ /00  Flying  Pilot:  O  Captain  O  FO 


Airport  Surface  Situation  Awareness  (ASSA) 

CDTI  Used  to... 

Cl 

C2 

C3 

C4 

Notes 

Y 

Y 

Y 

Y 

1.  ...determine  taxi  sequence? 

N 

N 

N 

N 

2.  ...evaluate  local  traffic 

Y 

Y 

Y 

Y 

situations? 

N 

N 

N 

N 

3 _ locate  targets  relative  to 

Y 

Y 

Y 

Y 

ground  reference  points  (e.g., 

N 

N 

N 

N 

taxiways)? 

N/A 

N/A 

N/A 

N/A 

CDTI  Features  (BLD2) 

Cl 

C2 

C3 

C4 

Notes 

1.  Use  TGT  (SEL)  to  identify 

Y 

Y 

Y 

Y 

assigned  taxi  interval? 

N 

N 

N 

N 

2.  Use  VEC  to  evaluate  target 

Y 

Y 

Y 

Y 

speed? 

N 

N 

N 

N 

Final  ADDroach  Runway  Occupancy  Awareness  (FAROA)  while  on  GROUND 

CDTI  Used  to... 

Cl 

C2 

C3 

C4 

Notes 

Y 

Y 

Y 

Y 

1 .  . .  .evaluate  runway  occupancy? 

N 

N 

N 

N 

2.  ...evaluate  final  approach 

Y 

Y 

Y 

Y 

corridor  for  possible  conflicts? 

N 

’  N 

N 

N 

3.  . .  .evaluate  departure  corridor  for 

Y 

Y 

Y 

Y 

possible  conflicts? 

N 

N 

N 

N 

CDTI  Features  (BLD2) 

Cl 

C2 

C3 

C4 

Notes 

Arc 

Arc 

Arc 

Arc 

Comp 

Comp 

Comp 

Comp 

1.  Which  display  mode  was  used 

Rose 

Rose 

Rose 

Rose 

most  frequently?  DSP  (ARC) 

(No  Arc/ 

(No  Arc/ 

(No  Arc/ 

(No  Arc/ 

No  Rose) 

No  Rose) 

No  Rose) 

No  Rose) 

2.  Use  Rt  and  Ri  to  adjust  range 

Y 

Y 

Y 

Y 

during  taxi? 

N 

N 

N 

N 

3.  Use  VEC  to  evaluate  target 

Y 

Y 

Y 

Y 

speed? 

N 

N 

N 

N 
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Departure  Spacing  (Events  1, 2,  and  3  - 

CDTI  Used  tn:. .  1  m 

-  Long  6.0  nm)  (Events  4  and  5  -  Short  4.5  nm) 

(  r** \  1  i  i 

1.  ...determine  separation  distance? 

Y 

Y 

Y 

Y 

Notes 

N 

N 

N 

N 

CDTI  Features  (BLD2) 

V-Cl-;:; 

C2 

C3 

C4 

Notes 

2,  Set  RR  to  separation  distance? 
(Long  =  6.0,  Short  =  4.5) 

Y 

Y 

Y 

Y 

(Build  4  ONLY) 

N 

N 

N 

N 

3.  Use  RT  and  RX  to  adjust  range 

Y 

Y 

Y 

Y 

prior  to  takeoff  roll? 

N 

N 

N 

N 

4.  Set  VEC  to  2  min  Long  or  1.5  min 

Y 

Y 

Y 

Y 

Short? 

N 

N 

N 

N 

5.  Use  LVL  (ALT)  to  set  altitude 

Y 

Y 

Y 

Y 

range  filter  for  departure  spacing? 

N 

N 

N 

N 

Initial  Approach  Spacing  (Events  1, 2,  and  3  -  Long  9.0  nm)  (Events  4  and  5  -  Short  7.0  nm) 


CDTI  Used 

1*  ...establish  and  maintain  separation 
distance? 

CDTI  Features  (BLD2) 

2.  Set  RR  to  separation  distance? 
(Long  =  9.0,  Short  =  7.0) 

3.  Select  TGT  (SEL)  to  identify 
target? 

4.  Set  VEC  to  2  min  Long  or  1.5  min 
Short? 


Cl 

Prc2 

Y 

Y 

N 

N 

■1?  CLP, 

C2 

Y 

Y 

N 

N 

Y 

Y 

N 

N 

Y 

Y 

Notes 


5.  Use  LVL  (ALT)  to  set  altitude 
range  filter  for  initial  approach 
spacing? 


6.  Use  VEC  to  evaluate  target  speed? 
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Final  Approach  Spacing  (Events  1. 2,  and  3  -  Long  5.0  nm)  (Events  4  and  5  -  Short  3.0  nm) 

CDTI  Used  to... 

Cl 

C2 

C3 

C4 

Notes 

1.  ...establish  and  maintain  separation 

Y 

Y 

Y 

Y 

distance?  i 

N 

N 

N 

N 

2.  ...evaluate  separation  by  pilot  not 

Y 

Y 

Y 

Y 

flying  ONLY? 

N 

N 

N 

N 

CDTI  Features  (BLD2) 

Cl 

C2 

C3 

C4 

Notes 

1.  Set  RR  to  separation  distance? 

Y 

Y 

Y 

Y 

(Long  =  5.0,  Short  =  3.0) 

N 

N 

N 

N 

Y 

Y 

Y 

Y 

2.  Use  TGT  (SEL)  to  identify  target? 

N 

N 

N 

N 

3.  Set  VEC  to  2  min  Long  or  1.5  min 

Y 

Y 

Y 

Y 

Short? 

N 

N 

N 

N 

Y 

Y 

Y 

Y 

4.  Use  VEC  to  evaluate  target  speed? 

N 

N 

N 

N 

Final  Approach  Runway  Occu 

pancy  Awareness  (FARO A)  while  in  AIR 

CDTI  Used  to... 

Cl 

C2 

C3 

C4 

Notes 

1.  ...evaluate  the  runway  environment 

Y 

Y 

Y 

Y 

for  possible  conflicts? 

N 

N 

N 

N 

CDTI  Features  (BLD2) 

Cl 

C2 

C3 

C4 

Notes 

L _ — _ 

1.  Use  Rt  and  Rl  to  adjust  range 

Y 

Y 

Y 

Y 

within  runway  environment? 

N 

N 

N 

N 

Arc 

Arc 

Arc 

Arc 

Comp 

Comp 

Comp 

Comp 

2.  Which  display  mode  was  used  most 

Rose 

Rose 

Rose 

Rose 

frequently?  ARC  (DSP) 

(No 

(No 

(No 

(No 

Arc/No 

Arc/No 

Arc/No 

Arc/No 

Rose) 

Rose) 

Rose) 

Rose) 

Y 

Y 

Y 

Y 

3.  Use  VEC  to  evaluate  target  speed? 

N 

N 

N 

N 

Runwav  Incursion  Scenarios  (Flight  Periods  1  and  2  ONLY) 

Notes 

- - - - - 

_ 1 1 — - - - - — “ - - - 

1.  Was  TRIOSl  Van  identified?  CDTI  ®©  Visual  ©©  No  O 

Distance  nm  (check  CDTI  for  distance) 

2.  Was  FAA  727  (N40)  identified?  CDTI  ©  ©  Visual  ©  ©  No  O 

Distance  nm  (check  CDTI  for  distance) 

B20 


Surface  Navigation 

CDTIUsed  to... 

C2 

C3 

C4 

Notes 

1. ..  .evaluate  the  airport  surface 

Y 

Y 

Y 

1 

environment  for  possible  conflicts? 

N 

N 

N 

KB 

2. ..  .identify  taxiways  on  aiiport 

Y 

Y 

Y 

— 

surface? 

N 

N 

N 

IS 

GDH  Features  (BLD2) 

Cl 

•:i-:C2 

C3 

C4 

Notes 

1.  Use  RT  and  Ri  to  adjust  range  on 

Y 

Y 

Y 

Y 

aiiport  surface? 

N  . 

N 

N 

N 

Arc 

Arc 

Arc 

Arc 

Comp 

Comp 

Comp 

Comp 

2.  Which  display  mode  was  used  most 

Rose 

Rose 

Rose 

Rose 

frequently?  ARC  (DSP) 

(No 

(No 

(No 

(No 

Arc/No 

Arc/No 

Arc/No 

Arc/No 

Rose) 

Rose) 

Rose) 

Rose) 

3.  Use  YEC  to  evaluate  target  speed? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

4.  Use  TGT  (SEL)  to  identify  target? 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

5.  The  Surface  Map  was  used  to 

O  Never 

O  Sometimes 

taxi... 

O  Always  O  N/A 

Taxi  Time 

Structured  Routes  ONLY 

Taxi  Speed 
(If  Available) 

Notes 

Location? 

Proximate  AC  ID? 

Outbound 

O  Structured 

O  Unstructured 

Start 

O  Correct 

O  Incorrect 

AC  Type 

Taxi  Speed 

Stop 

ACID 

Inbound 

Start 

AC  Type 

O  Structured 

O  Correct 

Taxi  Speed 

Stop 

O  Incorrect 

ACID 

O  Unstructured 
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Modified  Cooper-Harper  Workload  Scale 

Instructions:  Begin  in  lower-left  comer  and  follow  arrows  up  and/or  to  the  right.  Darken  the  circles  that  correspond 


Flight  Event:  ©  ©  ©  ©  © _ Aircraft  Tail  Number:  N _ Date:  10/ - /2000 


Full  Stop  1 : 
Full  Stop  2: 
Full  Stop  3: 

Pull  Ctnn  A  • 


OPNF 
OPNF 
OPNF 
n  pnb 


®©@©©©®®®® 

©®®©©©®©®® 

©@@©®®®®®® 


Difficulty  level 


Pilot  demand  level  Rating 


Very  easy, 
highly  desirable 

Mental  effort  is  minimal  and 
desired  CDTI  performance  is 
easily  attainable 

i 

Easy, 

desirable 

Mental  effort  is  low  and 
desired  CDTI  performance  is 
attainable 

2 

Fair,  mild 
difficulty 

Mental  effort  is  required  to 
attain  acceptable  CDTI 
performance 

3 

Is  the  CDTI 
satisfactory 
without 
improvement? 


Mental 
workload  is 
high  and 
should  be 
reduced 


Minor  but 

annoying 

difficulty 

Moderately  high  mental  effort 
is  required  to  attain  acceptable 
CDTI  performance 

4 

Moderately 

objectionable 

difficulty 

High  mental  effort  is  required 
to  attain  acceptable  CDTI 
performance 

5 

Very 

objectionable  but 
tolerable  difficulty 

Maximum  mental  effort  is 
required  to  attain  acceptable 
CDTI  performance 

6 

Is  adequate 
spacing 
performance 
attainable  with 
tolerable 
V  workload?  / 


Can  safe  use  of 
the  CDTI  for 
spacing  be 
accomplished 
most  of  the 
\  time?  / 


Major 

deficiencies, 
CDTI  redesign 
is  strongly 
recommended 


Major 

N  deficiencies, 

- ^  CDTI  redesign 

is  mandatory 


Major  difficulty 

Maximum  mental  effort  is 
required  to  bring  errors  to 
moderate  level 

7 

Major  difficulty 

Maximum  mental  effort  is 
required  to  avoid  large  or 
numerous  errors 

8 

Major  difficulty 

Intense  mental  effort  is 
required  for  acceptable  CDTI 
performance,  but  frequent  or 
numerous  errors  persist 

9 

Impossible 


Acceptable  CDTI  performance 
cannot  be  accomplished  reliably 


Pod  start  jctured  Interview 
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Flight  Event;  ©  ©  ®  ®  ® 

Aircraft  Tail  Number;  N _ 

Date;  10/ _ /2000 


Please  answer  these  questions  as  they  relate  to  the  following  OpEval-2  applications:  Departure  Spacing  [CDTI  Build 
2  and  4],  Approach  Spacing,  Airport  Surface  Situation  Awareness,  and  Final  Approach  Runway  Occupancy 
Awareness, 

1.  Asa  crew,  were  you  adequately  prepared  for  today's  mission? 


2.  In  general,  please  discuss  the  effect  of  the  MX-20/GX-60  equipment  on  your  interaction  with  ATC. 


3.  In  general,  please  discuss  the  effect  of  the  MX-20/GX-60  equipment  on  crew  roles  and  communication  (e.g., 
checklist  completion). 


4,  In  your  opinion,  what  rules  or  procedures  need  to  be  developed  and/or  implemented  to  support  the  efficient  use 
of  the  MX-20  for  OpEval-2  applications. 


5.  Did  you  make  a  maneuver  based  on  the  CDTI  only?  If  so,  why? 


6,  Are  there  any  general  or  specific  issues  (e,g.,  display,  control  panel,  etc)  about  the  MX-20/GX-60  equipment 
that  you  would  like  to  discuss? 


7.  Would  you  like  to  see  the  MX-20/GX-60  equipment  implemented  in  all  aircraft  that  you  fly? 


8,  Were  you  comfortable  with  your  knowledge  of  the  MX-20/GX-60  system? 


9.  Was  there  a  tradeoff  between  the  goal  of  increased  traffic  awareness  for  some  other  goal  or  task(s)? 


[CDTI  Build  2  and  4  ONLY] 

10.  What  information  should  be  available  to  ATC  to  support  Departure  Spacing  using  the  CDTI? 


Thank  you  for  your  time  and  candid  responses  to  our  questions. 
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